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ABSTRACT 

This report describes the background, selection, 
Planninq, conduct, and evaluation of the ten research training 
presessions, serving U02 educational researchers, conducted by the 
American Educational Research Association from 26 February to 2 march 
1970 prior to its annual meeting in Minneapolis. Topics of the 
sessions were 1) Survey Research in Education; 2) A Social 
Sy stems-Eield Studies Paradigm for Research on Organizational and 
Administrative Phenomena in Education; 3) S^tems Techniques in 
Counselor Fducation and Counseling Proqram Research; 4) Research and 
the Development of Instructional Theory; ?) Aoolied Linear Regression 
Analysis in Educational Research; 6) Human Pehavior Genetics and Its 
Implications for Educational and Behavioral Pesearch; 7 ) A Systems 
Aporoach to Instructional Research and Design; R) Person-Free Item 
Calibration and the Item-Tree Person Measurement; °) Multivariate 
Statistical Analysis in Educational Pesearch; 10) 

Statistical 'Techniques for Qualitative as well as 
Variables. Detailed descriptions for each session 
portion of the report. Evaluation results indicated t 
some participants were dissatisfied with the physical 
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INTRODUCTION 

During 26 February to 2 March 1970 the American 
Educational Research Association (AERA) conducted a 
program of ten research training presessions prior to 
the annual meeting of the Association in Minneapolis. 
The presessions served 402 educational researchers. 

The costs of the program were borne by AERA» the U. S. 
Office of Education, and the participants themselves. 
This report describes the background, selection, 
planning, conduct, and evaluation of the 1970 AERA 
Research Training Sessions. 



BACKGROUND OF THE RESEARCH TRAINING SESSIONS 

The 1970 presession programs can trace their 
origins to informal meetings of one or two days dura- 
tion involving a relative handful of selected researchers 
prior to the 1964 and 1965 annual AERA meetings. The 
1964 and 1965 informal meetings were not widely publi- 
cized and really did not have the training c ? researchers 
as their primary mission. However, they can he regarded 
as the precursors of the AERA research training ses- 
sions since in 1966 the prototypical "presession" was 
held as one of a group of three meetings in the tradi- 
tion of these previous preconvention meetings. The 
1966 presossion which set a pattern, thereafter adopted 
for the AERA presession programs, was a session dealing 
with experimental design under the direotion of Richard 
E. Shutz. This 1966 presession on experimental design 
was the first five-day presession sponsored by AERA and 
was , in addition, the first fdrmal research training 
program completed under Title IV of the Elementary and 
Secondary Education Aot of 1965. Further, the session 
direoted by Schutz was the initial presession in con- 
nection with a professional meeting that was systemati- 
cally evaluated with respect to the attainment of its 
objectives. 

Because of the suocess and aoceptanoe of the 1966 
presession or. experimental design, coupled with a growing 
interest of AERA members in the possibility of expanding 
and formaliking other presession meetings, AERA spon- 
sored a program of six courses in the 1967 presession 
program under the general chairmanship of Richard E. 
Sohutz* These sessions and their directors were the 
following: 

1. Bayesian Statistical Analysis 

Donald Meyer, Syracuse University 




2. Curriculum Research and Evaluation 

Robert L. Baker, Arizona State University 
W. James Popham, University of California, 
Los Angeles 

3. Design and Analysis of Comparative Experi- 
ments in Education 

Gene V. Glass, University of Illinois 

4. Educational Reoearch Management Procedures 

Desmond Cook, Ohio State University 

5. Multivariate Design and Analysis in Educa- 
tional Research 

Joo Ward, Southwest Educational Develop- 
ment Laboratory 

6. Research Strategies with Culturally Deprived 
Children 

Martin Deutsch, New York University 

The 1967 presession program was supported in part 
by a grant from the U. S. Office of Education under 
Title IV of ESEA, 1965. Approximately 500 rosearchers 
applied for the program and somewhat more than 300 
researchers actually participated. 

The response to the extended 1967 program was 
highly positive and led to a much expanded program of 
eleven presessions in 1968 under the chairmanship of 
Gene V. Glass. The 1968 sessions were the following: 

1. Research in Reading Instruction 

John R. Bormuth, University of Chicago 

2. Educational Research Management Procedures 

Desmond L. Cook, Ohio State University 

3. Anthropological Field Methodology in the 

Study of Education: With Particular Em- 

phasis on Classroom Behavior and Sohool 
Administration 

Frank W. Lutz, New York University 

4. Nonparametric Methods in Educational Research 

Leonard A. Marascuilo, University of 
California, Berkeley 

5. Design and Analysis of Comparative Experi- 
ments 

Jason Millman, Cornell University 

6. Evaluation: New Concepts in Scope, Strat- 

egy and Purposes 

C. Robert Pace, University of California, 
Los Angeles 
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7. The Computer and Natural Language 

Ellis B. Page, University of Connecticut 

8. Instructional Product Research 

W. James Pophara, University of California, 
Los Angeles 

Howard Sullivan, Southwest Regional Labora- 
tory for Educational Research and Develop- 
ment 

9. On-line Computer Applications in Educational 
Research 

Ronald 6. Ragsdale, Ontario Institute for 
Studies in Education 

.10. Multivariate Design and Analysis in Educa- 
tional Researoh 

Joe H. Ward, Southwest Educational Develop- 
ment Laboratory 

11. Developmental Processes in College Students 
Jonathan R. Warren, Educational Testing 
Service 

The 1968 presessions program was also supported 
in part by a grant from the U. S. Office of Education 
under Title IV of ESEA, 1965. Nearly 750 individuals 
applied for the program and approximately 550 actually 
participated. 

The 1969 sessions were the followings 

1. Research in Instructional Product Development 

Robert L. Baker 

2. Nonparametric Methods and Associated Post 
Hoc Procedures in Educational Research 

Leonard A. Maraacuilo 

3. The Computer and Natural Language 

Ellis B. Page 

4. Researoh on Methods for Improving Children's 
Learning Profioieney 

William D. Rohwer, Jr. 

5. Systems Approach in Counseling and Counselor 
Education 

T. Antoinette Ryan 

6. Multivariate Design and Analysis in Educa- 
tional Research 

Joe H. Ward, Jr. 
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7.. Anthropological Methods in Education Research 

Harry F. Wolcott 

8. Sample Free Test Calibration and Person Meas- 
urement in Educational Research 

Benjamin D. Wright 

9. Survey Research in Education 

James 6. Anderson 

10. Multiple Group Discriminant Strategy 

Paul R. Lohnes 

11. Bayesian Statistical Analysis 

Donald L. Meyer 

12. Design and Analysis of Comparative Experi- 
ments 

Kenneth Hopkins 

Jason Millman 

The 1969 presession program was also supported 
by the U. S. Office of Education with eight sessions 
held in Los Angeles prior to the annual meeting of AERA 
and with four sessions held in College Park* Maryland) 
following the annual meeting. Over 6S0 individuals 
applied for the programs and approximately SSO par- 
ticipated. 

THE 1970 RESEARCH TRAINING SESSIONS 

In the late spring of 1969 AERA president Roald 
F. Campbell appointed the following Research Training 



Leonard A. Marasouilo, Chairman 
University of California) 
Berkeley 

Gary E. Hanna) AF.RA Assistant 
for Professional Affairs 

Representative 

Frank Luts, Pennsylvania 
State University 

James MacDonald) University 
of Wisconsin 

Joe L. Byers, Michigan State 
University 

Calvin Dyer, University of 
Michigan 



Presessions Committee: 



Division 

A-Administration 

B-Curriculum and 
Objectives 

C-Learning and In- 
struction 

D- Measurement and 
Research Methodology 
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E-Student Development 
and Personnel Services 



Carl E. Thoresen, Stanford 
University 



F-History and 
Historiography 



Mark B. Beach, University of 
Rochester 



6-Social Context 
of Education 



Edward L. McDill, The Johns 
Hopkins University 



As in the past, it was considered advisable to extend 
the opportunity to propose a presession to any member 
of AERA (primarily by means of a call for proposals in 
the newsletter of the organization) and to solicit pro- 
posals from some 200-300 respected educational research- 
ers who might wish to propose a presession if invited. 

In accord with these suggestions, the following notice 
was placed in the April 1969 issue of the Education 
Researcher and in other journals read and used by edu- 
cational researchers. 



"Potential directors for the 1970 AERa Pre- 
sessions, to ba held in Minneapolis, Minnesota 
during the last week of February 1970, are being 
sought among the ranks of AERA members. The 
Presession Committee plans to place a proposal 
for approximately eight five-day Presessions in 
the hands of a granting agency in August. 

"Any AERA member interested in proposing and 
directing an AERA Presession for 1970 should 
write the Presession Committee Chairman, Leonard 
A. Marascuilo, for a proposal outline. Pro- 
posals are expected to be brief (no more than 
three or four pages) and tentative. Thera are 
no restrictions on content} it is hoped that a 
broad range of topics (non-methodological as 
well as methodological) will be proposed. The 
emphasis in the 1970 Presessions will once more 
be training , i.e., providing participants with 
specific competencies of relevance to their 
research activities. It is hoped that research 
methods relevant to the urban education problem 
can be represented in this year's Presessions. 
The deadline for receipt of proposals is July 
16, 1969. The Presession Committee will meet 
shortly thereafter to select those Presessions 
deemed worthy of support. 

"Request for proposal outlines and inquiries 
should be addressed to Leonard A. Marascuilo, 
4611 Tolman Hall, University of California, 
Berkeley, California, 94602." 

A similar call for proposals was sent to numerous out- 
standing researchers known to the members and chairman 
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of the Presession Committee . The formidable task of 
choosing only a portion of those proposals received 
fell to the Presessions Committee on July 24, 1969 at 
O' Hare International Airport in Chicago. The following 
individuals were in attendance: 



Division 



Representative 



A 



Dr. Frank Lutz 



B 



Dr. James MacDonald 



C Dr. Joe L. Byers 

D Dr. Calvin Dyer 

E Dr. Ray Hosford, University 

of Wisconsin, for Dr. Carl 
E. Thoresen 



F Dr. Mark B. Beach 

G Dr. Edward L. McDill 

Dr. Leonard A. Marascuilo, 
Chairman 

Mr. Gary E. Hanna, AERA Asst. 



Following is a summary of the meeting: 

The meeting commenced with a reaffirmation of the 
statement of training session purpose which had been 
approved by last year's Presessions Committee. The 
focus on training or disseminative sessions as opposed 
to semina l or generative sessions was particularly 
notecT It was agreed that in judging the proposals 
for AERA presentation, candor and professional honesty 
would be given high honor and that any comments of 
an evaluative nature made during the meeting would 
be treated with complete confidentiality. 

In addition, it was decided that because the quality 
of the presessions of previous years had been ex- 
cellent, every attempt would be made to maintain 
this excellence with the selction of the best pos- 
sible set of possible presessions. If two or more 
different groups submitted identical or similar 
proposals, and if the selection committee could not 
single out any one as the best, then the final de- 
cision would be based upon the known competencies 
of the director and instructional staff. It was 
the opinion of the members of the selection committee 
that fame as a researcher offered some valid mea- 
sures concerning the quality of the instruction to 
take place during the presessions; the intent being 
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that anyone who As a competent researcher is, by 
association, also a good teacher. 

Other business discussed and agreed to by the Com- 
mittee, but not directly pertinent to the proposal, was 
the decision to give the Committee continuity over 
time by the selection of the members on a rotating 
basis to a two year membership. The reason for this 
two years of service is to insure better proposal 
preparation for the following AERA presessions and 
to make future presessions more relevant to the needs 
of the members of the parent organization. In this 
sense, this decision is directly related to proposals 
that AERA will make to the U. S. Office of Education 
in subsequent years. 

In selecting the best possible set of presessions 
for the coming year, the following 10 criteria were 
used by each evaluator: 

Director and Tentative Staff 
T~. Experience and capability. 

2. Is the staff adequate for stated objectives? 
Content 

3^ Importance or need for topic. 

4. Appropriateness to presession format. 

Instructional Objectives 
3^ Clarity. 

6. Usefulness of competencies to be promoted. 
Anticipated Audience 

7^ Is audience appropriate for stated objectives? 

Tentative Schedule 
F! Extent of planning . 

Proposed Evaluation Activities 
5^ Extent of planning. 

Overall Rating 

10. Value to educational research. 

In addition, each evaluator gave the proposals an 
overall rating of: 

1. Reject 

2. Accept Conditionally 

3. Accept Unconditionally 

Using the overall ratings at the outset, several 
initial 'rounds' for evaluation were used to screen 
out thoso proposals considered inappropriate for one 
reason or another. The proposals surviving the first 
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and subsequent elimination rounds were then dis- 
cussed at considerable length. After several hours 
and the use of numerous ranking schemes > ten proposals 
were selected as worthy of support. 

The procedure agreed to at the meeting was to con- 
tact the ten directors by mail as soon as possible 
and inform them of the decision of the Committee. 

The ’rejectees' were sent airmail letters which 
described the Committee’s decision and the reason 
for that decision. In addition , the submitter of 
the proposal was thanked for his expression of in- 
terest and if the committee thought that the proposal 
deserved further consideration for next year's pro- 
gram, the submitter was given specific recommenda- 
tions to improve the proposal and/or was asked to 
correspond directly with the committee representa- 
tive of his division of AERA. 

The Committee also decided to produce its own eval- 
uation procedure for each of the presession. For 
. this evaluation, each member of the Committee agreed 
to draft an evaluation sheet which would be sent to 
the Chairman of the Committee. The Chairman would 
then produce a first draft of the evaluation sheet 
which would then be reviewed by each member of the 
Committee. They would then critique this first 
draft, send their recommendations to the Chairman 
who would then prepare a second draft. This second 
draft would be reviewed with the presession directors 
at the organizational meeting to be held in November 
1969. From their recommendations, a final evaluation 
sheet would be prepared. The final evaluation:, forms 
will be administered by an AERA representative in 
Minneapolis who will also be in charge of the hotel, 
meeting, and other arrangements required by the pre- 
session directors. Finally, suggestions were also 
made regarding presession topics for 1971. These 
suggestions are to be relayed to next year's Pre- 
session Committee Chairman. 

The AERA Research Training Presessions Committee was 
gratified at the response to its call for proposals and 
was pleased to have secured the services of the respected 
group of scholars who agreed to conduct research training 
sessions . 

The call for presession proposals published in 
the Educational Researcher plus personal notes from mem- 
bers of the Presessions Committee produced a total of 
21 proposals. The titles of the 11 proposals which were 
not accepted follow: 
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1. Research in Methods of Indexing for Users of Non- 

Book Educational Materials 

2. Students in High School: Research Approaches to 

Understanding 

3. Instructional Systems Design fcr Research 

4. Techniques in Programmatic Instructional Develop- 

ment 

6. Direct Observation as a Research Technique 

6. Organizational and Group Behavioral Theory in 

Educational Administration 

7. The evaluation of curricula: What to Evaluate and 

how to Evaluate . 

8. Piagetian Research and Mathematical Curriculum 

Research 

9. Foundations of Educational Evaluation 

10. Research in the Psychology of Curriculum 

11. Methods for Multivariate Data Analysis in Educa- 

tion and Psychology: Theory and Applications 

The titles of the proposals which were selected 
for presentation and the names of the directors are 
listed below: 

1. Survey Research in Education 

James G. Anderson, New Mexico State University 

2. A Social Systems-Field Studies Paradigm for 
Research on Organizational and Administrative 
Phenomena in Education 

Douglas R. Pierce, California State Polytechnic 
College 

3. Systems Research for Counselor Education, Coun- 
seling and Ifelated Research 

T. Antoinette Ryan, University of Hawaii 

4. Research and the Development of Instructional 
Theory 

Thomas J. Shuell, State University of New York 
at Buffalo 

5. Applied Linear Regression Analysis in Educational 
Research 

Joe H. Ward, Jr., Southwest Educational Develop- 
ment Laboratory 
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6. Human Behavior Genetics and Its Implications for 
Educational and Behavioral Research 

Steven G. VAndenberg, University of Colorado 

7. A Systems Approach to Instructional Research and 
Design 

Stephen L. Yelon, Michigan State University 
Roger 0. Scott, Southwest Regional Laboratory 
for Educational Research and Development 

8. Person-Free Item Calibration and Item-Free Person 
Measurement 

Benjamin Wright, University of Chicago 

9. Multivariate Statistical Analysis in Educational 
Research 

M, I. Charles E. Woodson, University of California, 
Berkeley 

10. Powerful Statistical Techniques for Qualitative 
as Well as Quantitative Variables 

Douglas A. Penfield, Rutgers University 
Maryellen McSweeney, Michigan State University 

It should be noted that five of the sessions are 
based directly on experience obtained in conducting 
previous presession training programs funded by the 
Office of Education. Effective training programs do 
not arise overnight. The 1970 sessions are making use 
of the beet of pre-1970 research training experience 
and the needs of the AERA membership. Some of the 
skills taught at some of the repeat presessions are 
invaluable to many educational researchers because of 
their basic connections to empirical research and 
investigations. The committee expressed the point of 
view that as long as a need for a specific topic exists, 
AERA should meet and attempt to satisfy this need. 

However, this should not be interpreted to mean that 
once a presession is approved it will continue to be 
approved each time the director submits a proposal. 
Instead, it should be interpreted to mean that if a 
research need continues and if no new group of re- 
searchers submits a similar and better proposal, then 
the committee will have a tendency to look with favor 
upon a repeat presession because of its importance to 
the members of AERA. 

After notification of acceptance, the 10 pre- 
session directors revised their proposals so that they 
were incorporated into an AERA proposal to the U. S. 

Office of Education. This proposal was subsequently 
approved for funding by USOE. 

A meeting was held on 25 November 1970 in Minnea- 
polis with all 10 training session directors present. 
Discussions of the forthcoming presessions centered 
around procedures for processing applications, in addi- 
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tion to many other procedural and substantive issues. 

The initial agreement concerned the method of proces- 
sing applications. The decision of the group, after 
considerable deliberation, was to set no deadlines for 
applications, but to judge applicants in the order in 
which they were received on the basis of each appli- 
cant's qualifications. In essence, the agreement reached 
was to employ a criterion-referenced rather than a norm- 
referenced scheme for accepting applicants. The step- 
by-step procedure for processing applicants to the 1970 
training sessions is described below. 



PROCEDURES FOR PROCESSING APPLICANTS TO 1970 AERA 

TRAINING SESSIONS 

1. All applications received by Professor Marascuilo 
were recorded and mailed to the first choice direc- 
tor. 

2. First choice director made an accept or reject 
decision within two or three days after receiving 
applications. The very first few applications which 
trickled in could be retained somewhat longer. 

a. If an applicant was accepted, the director 
sent the acceptance letter to the applicant 
and added applicant's name to a list of ac- 
cepted participants which was sent weekly 
to Professor Marascuilo. 

b. If the applicant was rejected, the applica- 
tion was sent directly to the second choice 
director or, if no second choice was given, 
it was sent back to Professor Marascuilo. 

3. The second choice directors also made accept or 
reject decisions within two or three days after 
receiving applications. 

a. If an acceptance was made, an acceptance 
letter was sent by the director, and the 
applicant's name was added to the lis." being 
sent weekly to Professor Marascuilo. 

b. If rejected, application form was returned 
to Professor Marascuilo. 

4. Letters informing all rejected applicants were 
sent by Professor Marascuilo. If certain direc- 
tors wished to do so, they sent ''sorry" letters 
also, but it was not necessary. Rejectees were 
invited to apply to other sessions in which they 
were interested. 
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5. Weekly totals of applicants accepted in all of the 
sessions were mailed to each director. 

6. Directors for whom large numbers of participants 
presented no instructional problem were urged to 
accept as many qualified applicants as possible. 



ANONYMOUS EVALUATIONS OF THE TRAINING SESSIONS 

This year an anonymous evaluation was made by 
the participants of the various presessions attended. 
This was achieved by means of a modified c .estionnaire 
used for the 1969 AERA Presessions. The questionnaire 
was administered by the Presession Director and assis- 
tants on the morning of the last day of the Presessions. 
The exact form of the questionnaire is as follows: 

AERA 1970 Research Training Sessions 

Participant Evaluation Form 

1. What was the name of your session? 

2. The overall quality of instruction in your session 
was : 

Excellent Good Average Fair Poor 

3. Leav*’g aside the quality of instruction for the 
moment, do you think the topic treated in your 
session should be included in next year's Pre- 
session Program? 

Yes No 

4. If you had it to do over again, would you apply 
for the session which you have just completed? 

Yes No 

5. If this same session is held again, would you 
recommend that others attend? 

Yes No 

6. Did you stay at a convention hotel At home 

With friends or relatives Other 

7. How much money did it cost to attend this session? 
How much of this cost will be reimbursed to you? 
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9. Where did you first learn about the training ses- 
sion? 

Educational Researcher Professional journal 
A one sheet announcement Other (Specify) 

9. How good was the scheduling and management of the 
Presession you attended? 

Very good Good Poor Very poor 

10. How good were the meeting room facilities for the 
Presession you attended? 

Very good Good Poor Very poor 

11. Do you think you had the appropriate prerequisites 
or prior knowledge to make what you learned at 
this Presession of research, teaching, or adminis- 
trative value to you? 

More than necessary Just the right amount 
Not enough 

12. Did the teaching staff make sufficient allowance 
for the variability in prior knowledge brought to 
the Presession by the participants? 

Nearly all of the time Most of the time 

Some of the time Hardly ever 

13. Was there sufficient time for you to interact with 
the staff with respect to information and knowledge 
presented in the Presession? 

Yes No 

14. Did you wish to discuss research problems that had 
arisen in your own work with staff members? 

Yes No 

If 'Yes,' was there an opportunity to pursue this 
interest? 

Quite a lot Some None at all 

15. Did the amount of work required by the staff of the 
participants seem acceptable to you? There was: 

Too much Just about right Too little 
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16. Will you use what you have learned in your own 
teaching or research in the immediate future? 

Yes No 

17. WLll you use what you have learned at a later 
date in future research or teaching? 

Yes No 

18. Would you like to learn more on the topic you 
studied here? 

Yes No 

19. Was five days a sufficient time to learn and 
master the material of your Presession? 

Yes No 

If ’No,' would you attend a seven day or two 
week Presession on the same material? 

No Yes, seven days Yes, two weeks 

20. Please write any additional comments and/or sug- 
gestions in the space below, or on the reverse 
side . 

In Table 1 are shown the percentage distribution 
of responses to the question related to the overall 
quality of the instruction for the ten presessions. As 
can be seen, most of the participants thought that the 
quality was excellent or good, with 87 percent of the 
respondents choosing these two response categories. 

Only 2 percent of the respondents reported that the 
instruction was poor. These statistics suggest that 
the selection committee was very fortunate in their 
choice of Presessions to support. 

In Table 2 are shown the responses to the ques- 
tion related to the desirability of including the at- 
tended Presession in the 1971 program. As can be seen, 
the participants are almost universally in agreement 
with the proposition to repeat the Presession attended. 
This is not too surprising since in every case, the 
topics and skills treated are of considerable value to 
educational research/ 

In Table 3 are shown the responses to the 
question relatea to the possibility of repeat atten- 
dance to the Presession attended. In this case 87 
percent of the participants reported that they would be 
willing to repeat the training experience. This sug- 
gests that the presession programs were of considerable 
value to the participants and that the instruction was, 
indeed, of high quality. 
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Table 1. The Overall Quality of Instruction 
in Your Session Was: Excellent, 

Good, Average, Fair, or Poor. 



Presession Number 



Response 
in % 


1 


2 


3 


4 


5 


6 


7 


8 


9 


10 


Total 


Excellent 


50 


6 


37 


38 


71 


22 


74 


50 


45 


60 


48 


Good 


41 


53 


45 


29 


26 


50 


19 


46 


42 


40 


39 


Average 


2 


29 


11 


19 


3 


22 


0 


0 


4 


0 


7 


Fair 


5 


6 


8 


10 


0 


0 


4 


4 


9 


0 


5 


Poor 


2 


6 


0 


5 


0 


6 


4 


0 


0 


0 


2 


Total 

Number 


44 


17 


38 


21 


38 


18 


27 


28 


55 


20 


306 



Table 2. Leaving Aside the Quality of Instruc- 
tion for the Moment, Do You Think the 
Topic Treated in Your Session Should 
be Included in Next Year's Presession 
Program: Yes or No. 



Presession Number 

Response 

in% 123456789 10 Total 



Yes 100 100 
No 0 0 
Total 

Number 44 16 



100 95 100 89 100 

0 5 0 11 0 

37 21 38 18 27 



96 1P0 100 99 

4 0 0 1 

28 56 20 305 



O 
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Table 3. If you had it to do Over Again, Would 
You Apply for the Session Which You 
Have Just Completed: Yes or No. 



Presesoion Number 



Response 
in % 


1 


2 


3 


4 


5 


6 


7 


8 


9 


10 


Total 


Yes 


82 


47 


97 


71 


97 


72 


85 


96 


89 


100 


87 


No 


18 


53 


3 


29 


3 


28 


15 


4 


11 


0 


13 


Total 

Number 


44 


17 


38 


21 


38 


18 


27 


28 


56 


20 


307 



Table 4. If This Same Session is Held Again, 

Would You Recommend That Other Attend: 
Yes or No. 



Presession Number 



Response 
in % 


1 


2 


3 


4 


5 


6 


7 


8 


9 


10 


Total 


Yes 


96 


77 


97 


86 


100 


83 


96 


96 


93 


95 


94 


No 


4 


23 


3 


14 


0 


17 


4 


4 


7 


5 


6 


Total 

Number 


44 


17 


38 


21 


38 


18 


27 


28 


55 


20 


306 



Table 5. Did You Stay at a Convention Hotel, 
At Home, With Friends or Relatives, 
or Other. 

Presession Number 

Response 



in % 


1 


2 


3 


4 


5 


6 


7 


8 


9 


10 


Total 


Hotel 


82 


94 


79 


86 


71 


61 


82 


82 


76 


95 


80 


Home 


9 


0 


11 


5 


24 


11 


11 


11 


7 


5 


10 


Friends 


2 


6 


8 


0 


3 


22 


4 


7 


6 


0 


5 


Other 


7 


0 


2 


9 


2 


6 


3 


0 


11 


0 


5 


Total 

Number 


44 


17 


38 


21 


38 


18 


27 


28 


55 


20 


306 
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Table 4 shows the responses to the question re- 
lated to whether a participant would recommend that 
others attend the presession if it were to be repeated. 
Again, there is an almost universal endorsement of the 
presession programs with 94 percnet of the participants 
indicating that they would recommend attendance by 
other researchers. A stronger endorsement of the 
program would be difficult to find. 

On the basis of these four questions, it can bv> 
concluded that the 1970 program was a definite success. 
The great majority of respondents felt the instruction 
was of exceptional quality, that the programs should 
be repeated, that repeat attendance was not a bad idea, 
and they they would recommend that others attend. 
Finally, it should be noted that these high praises 
extend across all ten presessions. 

The statistics reported in Table 5 concerning 
lodging during the Presessions shows that 80 percent 
of the participants stayed at a hotel in Minneapolis. 
Since 20 percent of the participants stayed at home or 
with relatives, the averages reported in Table 6 must 
be viewed with caution since they will underestimate 
the costs encountered by participants who came from 
areas outside of the St. Paul-Minneapolis metropolitan 
area. Excluding the information for Presession 9, 
which met in Chicago, it is seen that the average cost 
per participant ranged from a low of $154.28 to a high 
of $244.33. Of this amount, $99.03 to $202.59 was 
reimbursed, indicating that each participant had to 
spend about $40.00 to $60.00 of his own money to attend 
the presessions. For this reason, some thought might 
be given to the possibility of granting $50.00 to each 
participant to help pay the cost of attendance at 
future AERA Presessions. 

The statistics reported in Table 7 show how im- 
portant it is to advertise the presession program in 
journals other than the Educational Researcher . 
Approximately 41 percent of the participants learned 
of their Presession from other sources, with 22 percent 
reporting that they learned of the presession in other 
professional journals. Because of publication sched- 
ules, future directors should be urged to submit an- 
nouncements to professional journals as soon as possible 
so that the participant base can be expanded. 

According to the statistics reported in Table 8, 
about 88 percent of the participants thought that the 
presession attended was well scheduled and managed. 

Only two of the Presessions did not receive outstanding 
ratings on this dimension. Part of this poorer rating 
is most likely related to the content of the material 
and not to the inexperience or lack of interest on the 
part of the directors and teaching staff. 
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Table 6. How Much Money Did it Cost to Attend 
This Session? How Much of This Cost 
Will be Reimbursed to You? 



Presession 

Number 


Average Cost 
to Partici- 
pant 


Average Reimburse- 
ment to Partici- 
pant 


Total 

Number 


1 


$241.23 


$186.34 


44 


2 


228.76 


151.98 


17 


3 


203.79 


131.67 


38 


4 


215.43 


105.23 


22 


5 


154.28 


99.03 


39 


6 


219.17 


111.94 


18 


7 


244.33 


202.59 


27 


8 


206. r 


162.22 


28 


* 


166.84 


107.96 


57 


10 


202.50 


187.50 


20 



308 
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Table 7, Where Did You First Learn About the 
Training Sessions? Educational Re#* 
searcher . Professional Journal,' One 
kheet Announcement , or Other. 



Presession Number 



Response 
in \ 


1 


2 


3 


4 


5 


6 


7 


8 


9 


10 


Total 


Ed . Res . 


57 


77 


32 


71 


73 


39 


70 


41 


63 


70 


58 


Journal 


25 


23 


16 


24 


16 


33 


19 


26 


23 


20 


22 


One Sheet 


2 


0 


24 


0 


3 


17 


0 


4 


2 


5 


6 


Other 


16 


0 


28 


5 


8 


11 


11 


29 


12 


5 


14 


Total 

Number 


44 


17 


38 


21 


37 


18 


27 


27 


56 


20 


305 



Table 8. How Good Was the Scheduling and Manage- 
ment of the Presession You Attended: 
Very Good, Good, Poor, or Very Poor. 



Presession Number 



Response 
in % 


1 


2 


3 


4 


5 


6 


7 


8 


9 


10 


Total 


Very Good 


41 


18 


63 


38 


82 


33 


85 


46 


32 


80 


52 


Good 


41 


35 


29 


33 


16 


56 


11 


50 


54 


20 


36 


Poor 


16 


29 


5 


29 


2 


6 


4 


4 


13 


0 


10 


Very Poor 


2 


18 


3 


0 


0 


5 


0 


0 


1 


0 


2 


Total 

Number 


44 


1? 


38 


21 


38 


18 


27 


28 


56 


20 


307 
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As suggested by the statistics of Table 9. most 
participant's were pleased with the meeting room facil- 
ities. The notable exception to this satisfaction was 
expressed by the participants in the Presession directed 
by Dr. Vandenberg. Among his participants, 67 percent 
thought that the facilities were poor or very poor. 

At the same time, *42 percent of the participants in the 
Presession directed by Dr. Ryan also expressed the same 
degree of dissatisfaction. In both cases, the dis- 
satisfactions were justified. A number of avoidable 
complications developed at the hotel where these 
Presessions were being held. Poor records were main- 
tained by the catering staff and there was an indica- 
tion that the managers did not know that these Pre- 
sessions were to be held at their hotels even though 
correspondence was exchanged with the directors and 
the logistic staff in Minneapolis. In both cases, 
meeting rooms were changed during the middle of the 
Presession and a number of other unnecessary and un- 
called for interruptions were imposed upon the Pre- 
session by the management of the hotel. 

Some of these same sor'is of inconveniences were 
encountered by Dr. Yelon and Dr. Scott with respect 
to their Presession. However, their difficulties were 
minor when compared to those encountered by Dr. Ryan 
and Dr. Vandenberg. Finally, it should be noted that 
Presession 9, directed by Dr. Woodson, met at the 
University of Chicago Continuation Center. It appears 
that most participants were pleased with the facilities 
offered at the Center. This may suggest that future 
programs be considered for presentation at such cen- 
ters. In general, they are more suited and better pre- 
pared for educational programs than are central city 
hotels . 



According to the statistics reported in Table 10, 
about 20 percent of the participants thought that the 
prerequisite skills were higher than those possessed. 

This was especially true of Presession 9, directed by 
Dr. Woodson, and to a lesser degree of Presession 5, 
directed by Dr. Ward. This finding is not unexpected. 
Both of these Presessions assume a rather fundamental 
knowledge of statistical methodology and research 
sophistication. As will be seen, most of the partici- 
pants felt that these topics are best covered in a 
longer presession and for that reason it might be 
reasonable to schedule sessions similar to these for 
one week Instead of five days. It appears that if they 
were scheduled for a longer time period, educators and 
researchers would still attend so as to acquire the 
important skills taught in these sorts of Presessions. 

Finally, it should be noted that 71 percent of 
the participants in Presession 2, directed by Dr. Pierce, 
reported that their prior knowledge was more than that 
necessary for the presession topic. This may explain 
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Table 9 . How Good Were the Meeting Room 

Facilities for the Preseasion You 
Attended: Very Good, Good, Poor, 
Very Poor. 



Presession Number 



Response 
in % 


1 


2 


3 


4 


5 


6 


7 


8 


9 


10 


Total 


Very Good 


9 


47 


e 


14 


45 


0 


0 


4 


47 


10 


21 


Good 


61 


53 


50 


57 


55 


33 


50 


75 


49 


65 


55 


Poor 


27 


0 


32 


29 


0 


61 


42 


21 


2 


20 


21 


Very Poor 


3 


0 


10 


0 


0 


C 


8 


0 


2 


5 


3 


Total 

Number 


44 


17 


38 


21 


38 


18 


26 


28 


55 


20 


305 



Table 10 . Do You Think You Had the Appropriate 
Prerequisites or Prior Knowledge to 
Make What You Learned at this Pre- 
session of Research, Teaching, or 
Administrative Value to You? More 
Than Necessary, Just the Right Amount, 
Not Enough. 



Presession Number 



Response 
in \ 


1 


2 


3 


4 


5 


6 


7 


8 


9 


10 


Tot 


More Than 
Necessary 


25 


71 


16 


14 


13 


17 


30 


14 


4 


10 


18 


Right 

Amount 


61 


18 


71 


76 


63 


67 


70 


75 


46 


75 


62 


Not 

Enough 


14 


11 


13 


10 


24 


16 


0 


11 


50 


IS 


20 


Total 

Number 


44 


17 


38 


21 


38 


18 


27 


28 


54 


20 


305 
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the somewhat poorer, yet positive, showing given to 
this Presession on other questionnaire items. Es- 
sentially, this suggests that the Presession Selection 
Committee should guard against sponsoring programs 
that are not sufficiently advanced in materials to be 
covered and skills to be taught. 

According to the statistics of Table 11, the 
interest and flexibility of the instructional staff 
of the various presessions to the variability in prior 
knowledge possessed by the participants was quite good 
and was positively noted by the participants. Only 
4 percent of the participants stated that the staff 
hardly ever reacted to the variability in prior knowledge 
brought to the Presessions by the participants. The 
major exception of this evaluation was expressed by the 
participants in Presession 9, directed by Dr. Woodson. 

Of all the presessions given, it is this one that parti- 
cipants were least prepared for, and this is very un- 
fortunate for the advancement of educational research. 

In many respects, this was one of the most important 
presessions ever offered by AERA mainly because so 
many problems of education are multivariate in nature. 
Very few schools of education offer any sort of training 
in this particular subject area because of the relatively 
high degree of statistical sophistication required for 
its understanding and use. Because it is new and gen- 
erally difficult for most students, extreme flexibility 
is required on the part of the instructors. Therefore, 
if this program is offered again, as it should be, then 
special emphasis should be placed on achieving flexi- 
bility in the instructors so as to teach and communi- 
cate to the broadest group of participants possible. 

As indicated by the statistics of Table 12, 

87 percent of the participants reported that they had 
sufficient time to interaot with the staff. However, 
there are two notable exceptions and these were Pre- 
sessions 9 and 10, directed by Dr. Woodson and Drs. 
McSweeney and Penfield. For most of the participants, 
the subject matter of these Presessions was new and 
minimal familiarity was possessed by the participants. 
That they would have liked more interaction, more dis- 
cussion, and more assistance from the teaching staff 
is not unexpected. For this reason, it is suggested 
that presessions of this nature be programed for one 
week instead of five days. This would give staff and 
participants sufficient time to discuss the lecture 
material and research problems of interest to the 
participants . 

The importance of the presession subject matter 
to the participants is illustrated in Table 13. As 
can be seen, 73 percent reported that they would have 
liked to discuss their individual research problems 
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Table 11. Did the Teaching Staff Make Suf- 
ficient Allowance for the Varia- 
bility in Prior Knowledge Brought to 
the PreseBsions by the Participants: 
Nearly All of the Time* Most of the 
Time, Some of the Time, Hardly Ever. 









Presession 


i Number 










Response 
in \ 


1 


2 


3 


4 


5 


6 


7 


8 


9 


10 


Total 


Nearly All 
the Time 


25 


18 


35 


33 


26 


39 


33 


25 


6 


35 


25 


Most of 
the Time 


55 


24 


57 


14 


53 


50 


59 


43 


36 


45 


45 


Some of 
the Time 


18 


47 


8 


43 


21 


0 


7 


29 


53 


10 


25 


Hardly 

Ever 


2 


11 


0 


10 


0 


11 


1 


3 


5 


10 


5 


Total 

Number 


44 


17 


37 


21 


38 


18 


27 


23 


55 


20 


305 



Table 12. Was There Sufficient Time for You to 
Interact with the Staff With Respect 
to Information and Knowledge Presented 
in the Presession.* Yes or No. 



Pre8e8<*ion Number 



Response 
in \ 


1 


2 


3 


4 


5 


6 


7 


8 


9 


10 


Total 


Yes 


96 


100 


87 


95 


97 


94 


100 


89 


58 


75 


87 


No 


4 


0 


13 


5 


3 


6 


0 


11 


42 


25 


13 


Total 

Number 


44 


17 


38 


21 


3ft 


18 


26 


28 


55 


20 


305 
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with the staff members. The only exception to this was 
Presession 6, directed by Dr. Vandenberg. This excep- 
tion is not unusual since this particular Presession 
centered on imparting knowledge rather than improving 
the research skills of the participants. 

Among the participants who wished to discuss 
research problems with the staff, almost all had the 
opportunity. As reported in Table 14, only 2 percent 
of the participants said that they had none at all. 

As reported in Table 15, only 9 percent of the 
participants thought that the staff was unreasonable in 
the amount of work expected from the participants. This 
suggests that the directors gave special thought to 
what could reasonably be expected when teaching takes 
place in the non-ideal conditions that generally pre- 
vail in a hotel. 

As indicated by the responses of Table 16, 
about 90 percent of the respondents see immediate 
usage or application of the materials learned at the 
presession attended. This same degree of usage is 
also indicated in Table 17, where it is seen that 98 
percent report that they expect to use this new acquired 
skill at a later time. 

The immense value of the presession selected 
for presentation to the participants is indicated in 
Table 18, since it is seen that 98 percent of the par- 
ticipants reported that they would like to learn more 
on the topic studied in the presession attended. 

As reported in Table 19, only 37 percent of the 
participants thought that 5 days was sufficient time to 
learn and master the material of the presession attended. 
This was especially true for Presessions 5, 9, and 10 
which were highly statistical in nature. 

Among those respondents who felt that five days 
were not sufficient time for coverage of presession 
materials, 79 percent reported that they would attend 
a presession of seven days' length. In fact, as 
shown in Table 20, 42 percent reported that they would 
be willing to attend a two week presession. 

In summary, it can be definitely concluded that 
the 1970 Presessions met the expectations of the parti- 
cipants. They reported that the instruction was ex- 
cellent, there were opportunities of discussing re- 
search problems, and that the material learned would 
eventually make its way into their research. Finally, 
they reported that they would like to learn more about 
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Table 15. Did the Amount of Work Required by 

the Staff, of the Participants, Seem 
Acceptable: There was: Too Much, 

Just About Right, Too Little* 



Presession Number 



Response 
in % 


1 


2 


3 


4 


5 


6 


7 


8 


9 


10 


Total 


Too Much 


7 


0 


8 


0 


16 


6 


4 


0 


19 


15 


9 


Just Right 


82 


59 


90 


71 


84 


83 


93 


96 


73 


80 


82 


Too Little 


11 


41 


2 


29 


0 


11 


3 


4 


8 


5 


9 


Total 

Number 




17 


38 


21 


38 


18 


27 


28 


52 


20 


303 



Table 16. Will You Use What You Have Learned in 
Your Own Teaching or Research in the 
Immediate Future: Yes or No. 



Presession Number 



Response 
in \ 


1 


2 


3 


4 


5 


6 


7 


8 


9 


10 


Total 


Yes 


96 


88 


95 


95 


84 


89 


93 


96 


76 


100 


90 


No 


4 


12 


5 


5 


16 


11 


7 


4 


24 


0 


10 


Total 

Number 


44 


17 


38 


21 


38 


18 


27 


28 


S4 


20 


305 



Table 17. Will You Use What You Have Learned at 
a Later Date in Future Research or 
Teaching: Yes or No. 

PreseBsion Number 



uv: i o w 

in \ 


1 


2 


3 


4 


5 


6 


7 


8 


9 


10 


Total 


Yes 


100 


94 


97 


95 


100 


94 


96 


100 


98 


100 


98 


No 


0 


6 


3 


5 


0 


6 


4 


0 


2 


0 


2 


Total 

Number 


44 


17 


38 


21 


38 


18 


27 


28 


55 


20 


306 
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Table 18. 


Would 


You Like to Learn 


More 


on 


the 






Topic You Studied Here: 


Yes 


or 


No. 






Presession Number 








Response 
in % 


1 2 


3 


4 S 6 7 8 


9 


10 


Total 


Yes 


96 100 


95 


95 100 100 100 100 


100 


100 


98 


No 


4 0 


5 


5 0 0 0 0 


0 


0 


2 


Total 

Number 


44 17 


38 


21 38 18 26 28 


55 


20 


305 



Table 19. Was Five Jays a Sufficient Time to 

Learn and M aster the Material of Your 
Presession: Yes or No. 



Presession Number 



Response 
in \ 


1 


2 


3 


4 


5 


6 


7 


8 


9 


10 


Total 


Yes 


43 


50 


57 


47 


13 


71 


56 


61 


5 


20 


37 


No 


57 


40 


43 


53 


87 


29 


44 


39 


95 


80 


63 


Total 

Number 


44 


16 


37 


19 


38 


17 


27 


28 


56 


20 


302 



Table 20. If Answer to Question in Table 19 
Was 'No,' Would You Attend a Seven 
Day or Two Week Presession on the 
Same Material: No» Yes Seven Days* 

Yes Two Weeks. 



Presession Number 



Response 
in S 


1 


2 


3 


4 


5 


6 


7 


8 


9 


10 


Total 


No 


21 


38 


25 


44 


16 


0 


0 


14 


19 


35 


21 


Seven Days 


54 


25 


31 


22 


38 


80 


58 


67 


25 


29 


37 


Two Weeks 


25 


37 


44 


34 


46 


20 


42 


29 


56 


36 


42 


Total 

Number 


24 


8 


16 


9 


32 


5 


12 


7 


52 


17 


182 




28 . 



the topics covered and that they would be quite willing 
to attend presessions that extended over one week. In 
essence, this suggests that the Presession Selection 
Committee should be congratulated for their final 
selection. Without doubt, the 1970 Presessions were 
well received by the educational and behavioral re- 
searchers who attended sessions in Minneapolis and 
in Chicago. 

In the remainder cf this report, descriptions 
of the 10 training sessions are presented with each of 
the following elements being included: 

1. Title 

2. Staff 

3. General Description 

4. Objectives 

5. Schedule 

6. Participants 

7. Instructional and Evaluation Materials 

8. Evaluation and Test Results 

9. Director's Evaluation 

These descriptions are drawn largely from reports 
supplied by the directors of the 10 sessions. Minor 
editing was undertaken to preserve some degree of uni- 
formity in the reports. 




TRAINING SESSION DESCRIPTIONS 
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PRESESSION I 



1. Title: Survey Research in Education 



2. Staff: James G. Anderson 

(Director) 



New Mexico State 
University 



Harry R. Potter 
Harlley E. McKean 

Stanley Ball 

Dean Naf&iger 



Purdue University 

New Mexico State 
University 

New Mexico State 
University 

New Mexico State 
University 



3. General Description: 



Presession Number 1 was a special training ses- 
sion in Survey Research techniques aimed at demonstrating 
the use of the survey as a primary instrument for 
scientific study in education. The program was designed 
for those with little or no previous experience with 
Survey Research and covered all phases of Survey Re- 
search including, the design of a survey, sampling, 
construction of scales and indices, questionnaire con- 
struction, interviewing techniques, coding of data, 
and methods of analysis. 



4. Objectives: 

By completing a large number of class exercises 
with actual survey data collected by the participants 
on themselves, participants acquired a practical working 
knowledge of all aspects of survey research. As a 
result of attendance at this presession, participants 
should be able to: 

1. Plan a survey in sufficient detail to answer 
questions regarding: 

a. the objectives of the survey 

b. the population to be studied 

c. the means to be used in selecting a sample 

d. the data that is to be collected 

e. the methods to be used to collect these 
data 
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f. the approach to be used in recruiting* 
training, and supervising field workers 

g. the methods to be used in editing, coding, 
and tabulating the survey data 

h. the analytical techniques to be used 

i. the reports that are to be prepared 

j . the cost of the survey 

k. the scheduling of the survey 

2. Define the survey population, ascertain the 
necessary sample size to obtain the desired accuracy, 
choose a selection process, select a probability sample, 
and compute sample estimates of population values for 
relatively simple types of surveys. 

3. Utilize certain scaling techniques such as 
the Likert scale, the method of equal appearing in- 
tervals, and Guttman scaling to develop scales and 
indices from survey data. 

4. Design questionnaires and interview schedules. 

5. Conduct interviews and train staff members 
to conduct interviews. 

6. Recruit, train, and supervise staff for 
survey field work. 

7. Edit, code, and process survey data in pre- 
paration for analysis. 

8. Analyze survey data by utilizing a library 
of computer programs to: 

a. examine the distribution of a single 
variable and its numerical properties 

b. analyze the relationship between two 
qualitative variables by means of a 
cross-tabulation 

c. compute selected measures of association 
between two variables 

d. analyze the relationships among three or 
more variables by cross-tabulations 

e. compute partial measures of association 
among three or more variables 
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f. perform correlation and multiple regres- 
sion analysis. 



5. Schedule: 

A questionnaire was completed by each participant 
and coded on the morning of the first day of the training 
session in order to provide data for the class exer- 
cises. This was followed by a planning exercise in 
the afternoon. A review of statistics was held in the 
evening of the first day prior to the discussion and 
class exercises on sampling. 

Sampling and measurement were treated on the 
second day. Participants worked through the first two 
exercises on sampling in order to gain familiarity 
with simple random sampling and cluster sampling. Data 
for these exercises were generated from decks of num- 
bered cards that were provided each student. 

Guttman scaling was used to initiate that part 
of the program dealing with measurement. After partici- 
pants had scaled data from the code sheets using the 
Guttman technique, other types of scaling were discussed 
and illustrated. 

The following methods of collecting survey data 
were discussed and illustrated on the third day of the 
session: 

1. Use of archives 

2. Direct observation 

3. Questionnaires 

4. Interviewing 

The remainder of the third day and the fourth 
day were devoted to the analysis or survey data from 
the coding forms. Participants worked through and dis- 
cussed the methodology involved in class exercises. 

On the morning of the final day of the session 
a large number of graphical methods that can be used 
to display survey data were demonstrated. This was 
followed by a general review of the entire process of 
conducting a survey. Finally, participants were asked 
to complete the AERA evaluation form and to write down 
any comments, suggestions, or criticisms concerning the 
training session. 
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First Day: Morning: 

1, Introduction to Survey Research 
Administration and Coding of Class Ques- 
tionnaire 

Afternoon: 

2 Planning Surveys 
Survey Case Studies 

Evening: 

Review of Statistics or PERT 

Second Day: Morning: 

3. Sampling 

Afternoon: 

4. Measurement 



Evening: 

5. Data Collection 
Questionnaire 

Third Day: Morning: 

5. Data Collection 
Interviewing 

5. Data Collection 
Coding 

Afternoon: 

6. Data Analysis I 

7 . Data Analysis II 

Evening: 

7. Data Analysis II 

Fourth Day: Morning: 

8. Data Analysis III 

Afternoon: 

9. Data Analysis IV 

Fifth Day: Morning: 

Graphic Presentation of Survey Data 
Evaluation of Training Session 



6. Participants: 

Of the 55 participants, 39, or 71 percent, were 
male. They came from 29 states with 47 percent coming 
from the Midwestern states of Illinois, Indiana, Iowa, 
Kansas, Michigan, Minnesota, Ohio, and Wisconsin. There 
was one participant who came from California, while 5 
of the participants were from Canada. Sixty-two per- 
cent of the participants were employed at a University 
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or College. The remaining 38 percent came from junior 
colleges, public school systems, state departments of 
education, or private corporations. Approximately one- 
third of the participants had attended other AERA Pre- 
session in previous years. Finally, 64 percent of the 
participants were holders of a doctorate. 

7. Instructional and Evaluation Materials: 

Each participant was provided with a notebook 
and a copy of Sociological Analaysis : An Empirical 

Approach Through Replication by Murray A. Straus and 
Joel I. Nelson (New York: Harper and Row, 1968). 

As the Table of Contents fromthe notebook in- 
dicates, instruction and evaluation were broken down 
into ten sections. Data for the exercises were ob- 
tained by having each participant complete the ques- 
tionnaire from kelson and Straus. Participants then 
exchanged questionnaires and coded them on the coding 
form. The coding forms were collected and 20 sets of 
the 65 forms were reproduced so that a set was avail- 
able for each group of three persons. 

An optional session was held to review elemen- 
tary statistics on the first evening of the session 
by Dr. McKean. The materials that he prepared for 
this review are contained in the Appendices, Section 
10 of the notebook. 

Notebook Contents: 

1. Introduction 

A. Course Outline 

B. Survey Research in Education: The Case of 

a Misconstrued Technique by Sam D. Sieber 

2. Planning Surveys 

A. Planning Surveys 

Exercise 1: Planning a Survey 

B. Scheduling Survey^-PERT 

Exercise 2: Scheduling a Survey 

3. Sampling 

A. Sampling by Philip J. McCarthy 

B. Sampling 

Exercise 3: Simple Random Sampling 

Exercise 4. Stratified Sampling 



O 
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4. Measurement 



Scaling 



Exercise 


5 : 


Measurement of Socio-Economic 
Status 


Exercise 


6: 


Social D istance Scale 


Exercise 


7: 


The Method of Equal-Appearing 
Intervals 


Exercise 


8: 


Guttman Scaling 



5. Data Collection 



A. 



B. 



C. 

D. 

E. 



F. 



G • 



H. 



Questionnaire 

Exercise 9: Class Questionnaire 

Questionnaire Construction 
Exercise 10: Planning a Survey 

Exercise 11: Design of a Questionnaire 

Exercise 12: Administering a Questionnaire 

Interview Schedule 

The Questionnaire and the Interview 
Interview Schedule Construction 
Exercise 13: Planning a Survey 

Exercise 14: Design of an Interview Sched- 

ule 

Exercise 15: Interviewing 

The Observation and Recording of Behavior 
Exercise 16: The Observation and Recording 

of Behavior 

Coding Instructions: What People Like 

and Dislike About the Schools 
Coding and Tabulation of Survey Data 
Exercise 17: Coding and INdex Construction 

Exercise 18: Development of Codes from 

Empirical Data 

Exercise 19: Coding Data for Transfer 

to Punched Cards 

Exercise 20: Planning for the Tabulation 

of Survey Data 



6. Data Analysis I 

Description and Summary of Single Variables 
Exercise 21: The use of Percentages 

Exercise 22: The Use of Means 

Exercise 23: The Use of Percentages 



7. Data Analysis II 



Relationships Between TWo Variables: 
of Percentages 



The Use 



Exercise 24: 
Exercise 25: 
Exercise 26: 



The Use of Percentages 
The Use of Percentages 
The Use of Means 



O 
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8. Data Analysis III 



Relationships Between Two Variables: 
of Association 



Measures 



Exercise 27: 
Exercise 28: 
Exercise 29: 
Exercise 30: 
Exercise 31: 

Exercise 32: 



9. Data Analysis IV 



The Percentage Difference 

Chi Square 

Gamma 

Product Moment Correlation 
Partitioning the Degrees of 
Freedom in Contingency Tables 
Testing for Trends in Con- 
tingency Tables 
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Multivariate Analysis: 
Several Variables 
Exercise 33: 
Exercise 34: 
Exercise 35: 
Exercise 36: 
Exercise 37: 
Exercise 38: 
Appendices 



Relationships Among 



Percentages 

Percentages 



The Use of 
The Use of 
Gamma 

Product Moment Correlation 

The Use of Means 

Multiple Regression Analysis 



A. Review of Elementary Statistics 

B. Tables 

C. Research Design and Analysis in the Be- 

havioral Sciences: An Annotated Biblio- 

graphy by James G. Anderson 



8. Evaluation: 



Participant evaluation was based upon the comple- 
tion of 38 different class exercises. The proportion 
of students who completed each exercise is shown in the 
following table: 



Exercise Subject 



% completing 



1 Planning a survey 100% 

2 Scheduling a survey** 40 

3 Simple Random Sampling 100 

4 Stratified Random Sampling 100 

5 Measurement of socioeconomic status 100 

6 Social distance scale 100 

7 Method of equal appearing intervals 100 

8 Guttms.n scaling 100 

9 Class questionnaire 100 

10 Planning a survey NA 

11 Design of a questionnaire 100 

12 Administering a questionnaire NA 

13 Planning a survey NA 

14 Design of an interview schedule 100 

15 Interviewing NA 
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Subject 



% com- 
pleting 



Exercise 



16 


The observation and recording of 






behavior 


NA 


17 


Coding and index construction 


100% 


18 


Development of codes from empirical 


100 




data 


19 


Coding data for transfer to cards 


NA 


20 


Planning tabulation of data 


NA 


21 


Use of Percentages 


100 


22 


Use of Means 


100 


23 


Use of Percentages 


100 


2k 


Use of Percentages 


100 


25 


Use of Percentages 


100 


26 


Use of Means 


100 


27 


Percentage difference 
Chi Square 


NA 


28 


100 


29 


Gamma 


100 


30 


Product Moment Correlation 


NA 


31 


Partitioning Degrees of Freedom 


NA 


32 


Testing for Trends 


NA 


33 


Use of Percentages 


NA 


34 


Use of Percentages 


100 


35 


Gamma 


NA 


36 


Product Moment Correlation 


NA 


37 


Use of Means 


NA 


38 


Multiple Regression Analysis 
Review of Statistics** 


NA 


1 


Descriptive Statistics 


50 


2 


Random Variables 


50 


3 


Hypothesis Testing 


50 


4 


Comparing Two Means 


50 


5 


Use of Chi Square 


50 


6 


Uses of the F Distribution 


50 



**0ptional Exercise 
NA Exercise not assigned 

9. Director's Evaluation: 

In all, 73 persons applied and were accepted 
for Presession 1. Of these applicants only one-third 
had participated in a previous AERA sponsored Research 
Training Program. The large number of first-time 
applicants for this training session would suggest 
that many persons involved in teaching and performing 
educational research desire an opportunity to learn 
about survey research methodology, and were not ex- 
posed to these techniques during their academic train- 
ing. 



This is borne out by analysis of the backgrounds 
of the 55 men and women who participated inthe session. 
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Two-thirds of the participants hold the doctorate. The 
median number of courses in statistics and experimental 
design completed by these individuals is between 3 and 
4. However, almost all of the applicants mentioned 
their lack of formal training in survey research method- 
ology and their increased involvement with research 
and educational policy problems requiring survey tech- 
niques as the major reason for applying for this ses- 
sion. It would appear that most participants followed 
a traditional academic program in which they were 
exposed to several courses in statistics and experi- 
mental design with little or no exposure to survey 
research methodology. 

The hotel setting for these presessions, I must 
say, leaves a great deal to be desired. The Leamington 
over-scheduled events for their existing facilities 
necessitating a good deal of shifting from room to room 
the first two days. Also there were major problems of 
security in such a setting due to the personal belongings 
that had to be carried by participants during the meal 
breaks, etc. Finally, the difficulties of finding 
reasonable restaurants, close to the hotel where par- 
ticipants can be served promptly played havoc with the 
schedule at times. 

A university campus or center, such as the 
Center for Adult Education at the University of Mary- 
land, provided a find setting for the post sessions 
last year. Moreover, the staff of such centers provide 
excellent support both prior to and during the sessions. 
Having a University handle registration and logistics 
as well as the avialability of AV equipment, duplica- 
tion facilities, seminar rooms for group work, etc., has 
much to recommend it for the future Research Training 
Sessions . 

The session itself went quite well, in my esti- 
mation. We accomplished a great deal of what we had 
intended to accomplish. However, several of the 
participants' suggestions that were outlined earlier 
are important for future training sessions based on 
this topic. For example, making available a computer 
terminal for the session next time would serve to in- 
troduce participants to the very important technical 
aspects of processing survey data. At the same time use 
of such a terminal would cut down on the tedious and 
time consuming clerical operations and calculations 
that are necessary when analyzing data. We had planned 
to use a terminal this year, but the hotel setting made 
it virtually impossible to do so. 

Although we did attempt to differentiate the 
program to suit the interests and level of participants 
where ever possible, such differentiation might easily 
be incorporated into the program and notebook from the 
very beginning by providing a series of optional and 
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supplementary exercises for participants who wish to 
pursue special topics and/or more sophisticated re- 
search methodologies such as the use of multivariate 
statistical techniques in the analysis of survey data 
and a variety of sophisticated scaling techniques 
such as factor analysis* etc. Additional material on 
a wider variety of data collection techniques could 
easily be included in the notebook. In this fashion 
groups of participants could easily simultaneously work 
on different exercises aimed at different levels but all 
are related to the same central topic. Also* with as 
large a group of partcipants as we had both the times 
the training sessions has been offered, it would be 
wise to add additional staff next year in order to 
provide for more individual assistance. Again, a 
number of participants mentioned this in their comments 
and suggestions. 

In summary* I believe that the Research Training 
Session on Survey Research proved to be most popular 
and successful and should be offered again next year. 

In fact, the persons who applied for this session last 
year and this year but who did not attend for a variety 
of reasons may well welcome an opportunity to participate 
the next time that the topic is offered if their ex- 
pressions of regret at not being able to attend are 
any indication. 
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PRESESSION II 



1 . 



2 . 



3. 



Title: A Social Systems-Field Studies Paradigm for 

Research on Organizational and Administra- 
tive Phenomena in Education 



Staff: Douglas R. Pierce 

(Director) 

Dan Lortie 

Erwin Miklos 

Philip Runkel 

James D. Thompson 

General Description: 



California State Poly- 
technic College 

University of Chicago 

University of Alberta 

University of Oregon 

Vanderbilt University 



Increasing attention has been directed in re- 
cent years to the development of social systems con- 
ceptualizations, 6 their utility in generating under- 
standing of school organizations, and to methodological 
implications of attempting to understand schools as 
social systems. A primary purpose of this presession 
was to increase social systems-f ield studies paradigm 
consensus; that is, to increase the liklihood that 
participants would respond in social systems-field 
studies terms to basic questions: What are tha impor- 

tant questions about the structure and processes of 
organizational behavior in education? What are the 
effective ways of finding answers to those questions? 
What are satisfactory criteria for acceptance of the 
answers? Content was concerned with such questions as 
the following: What are the properties and relation- 

ships which are considered critical in social systems^ 
theories? What distinctive properties and relationships 
are observeable in school organizations? What are 
some of the research techniques being utilized in 
studies of schools as social systems? What answers 
are resulting from such studies? What are the critical 
issues raised by approaching schools as social systems, 
susceptible to understanding and to development through 
field studies techniques? 



4. Objectives: 

The major objective was to increase participants' 
abilities to design and implement social systems-field 
studies of school organizations. Participation in the 
presession was expected to increase the likelihood of: 

1. Asking researchable questions for which at 
least partial answers could be anticipated through 
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social systems theory, and for which the answers 
could be expected to be important in developing a 
comprehensive understanding of school organizations. 

2. Designing and implementing field studies of 
school organizations, such studies to include com- 
parative and multivariate designs and to utilize 
multiple observational systems. 

3. Increased grappling with critical issues 
of the interaction of theory and technique, stan- 
dards for judging the adequacy of research answers; 
further, increased interaction and communication 
with colleagues both about such issues and about 
studies in which they become involved. 

5. Schedule: 

First Day: Session 1: 

Orientation: statement of objectives, 

introduction of staff with identification 
of their special competencies and interests, 
small group introduction of participants- 
large group summation 

Session 2: 

Systems theory in organizational research, 
traditional preoccupation with core tech- 
nology/closed system tradition in adminis- 
tration studies; scale expansion, increased 
technological complexity, and task environ- 
ment redefinition; interdependence and 
contingency; systems transactions; induce- 
ments-contributions model; predictions as 
to directions of organizational growth 
and compensations for necessity of dealing 
with uncertainty 

Session 3: 

Inventory of research activities underway 
or being designed by participants: for- 

mation of research design sub-groups 

Second Day: Session 1: 

Techniques for developing and understand- 
ing organizational problem-solving in a 
school faculty: intervention as field 

experimentation; reactivity in data 
gathering; multiple observational tech- 
niques 

Session 2: 

Research design activity in sub-groups: 
issues of macro and micro orientation; 
development of problem statements; speci- 




Third Day: 



Fourth Day: 



Fifth Day: 
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fication of methodological issues; large- 
group summation and feedback 

Session 1: 

Realistic working models of school organi- 
zations: specification of distinctive 

properties and relationships of school 
organizations; utility of an 'as if' 
perspective in research design; essential- 
ness of comparative design in field 
studies 

Session 2: 

Development of research studies in sub- 
groups: f ocu8 upon problems of sampling 

in field studies! data collection tech- 
niques i presentations on interviewing! 
participant observation! and reports of 
research on elementary school teachers 

Session 1: 

Strategies for getting on; organizing 
ideas about organizations to increase 
the likelihood of their utilization in 
field studies of school organizations: 
inputs t transformations i output* and feed- 
back; boundaries and environmental varia- 
bles; vertical and lateral differentia- 
tion 

Session 2: 

Critique of research on schools in the 
bureaucratic tradition; specification and 
illustration of five specific alternatives 
for guiding research. Criterion problems 
in field studies of school organizations. 
Hypotheses about organizational transfor- 
mation! domain expansion and task environ- 
ment expansion! and increasing frequency 
of intensive technology organizations 

Session 1: 

Challenges of an open systems orientation 
for research on schools: multiplication 

of value issues ! informational demands % 
interdependences; non-zero-sum alterna- 
tives; incremental problem solving pro- 
cesses; transformation of 'norms of ration- 
ality* toward 'Parento Optimum;' redefini- 
tion of coordination; 'muddling through' 
strategies/development of collegiums; 
intervention strategies for development 
and research; appropriateness of schools 
for intervention and transformation toward 
intensive technology model 
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Session 2: 

Critique of social systems conceptuali- 
zation as grounding for research design 
and of field study techniques for develop- 
mental and for research effectiveness*, 
planning for carrying forward work on 
field studies instigated during the pre- 
session; critique of presession activity 
and immediate outcomes 

6. Participants: 

The age of the thirty participants ranged from 
29 to 55, with the average age being 38.7. Twelve 
participants were older than forty, and of those, five 
were 50 or more years of age; six participants were 
younger than 35, with twalve between 35 and 40 years 
of age. Only one of the participants was female. 

Most participants (23 out of 30, or 77 percent) had 
not previously attended an AERA presession. 

Twenty-three participants (77 percent) were 
from a college or university. Regional Education 
Laboratories contributed three persons, and a stat-a 
education department, a city mental hygiene clinic, 
and a professional association each contributed one 
person. Most participants (19) were from the midwes- 
tern portion of the nation. However, 7 were from the 
southern portion, 5 were from the eastern portion, 

3 from the western portion, and 1 was from Canada. 

Eighty percent of the participants held the 
doctorate; the balance were candidates for the doctor- 
ate. Major fields of study were diversified: 47 per- 

cent listed educational administration, 27 percent 
listed psychology (five educational psychology, and 
one each in social psychology and general psychology), 
10 percent listed sociology (including sociology of 
education), and the balance were distributed among 
educational research, teacher preparation, and inter- 
national education. The average number of articles 
reported was 7.4, with two persons reporting more than 
30, and two reporting none. The average of funded 
projects was .8, with sixteen persons reporting no 
funded projects. 

In response on a seven-point scale to the ques- 
tion, "How knowledgeable are you of social systems 
concepts?" the mean response was 3.6 for the 23 partici 
pants who responded to the survey prior to arrival. 
Similarly, the mean response was 3.4 to the question, 
"How knowledgeable are you of field studies techniques? 
For both items, responses clustered near the extremes 
of the scale; participants tended to report being 
either 'naive* or relatively 'sophisticated' rather 
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than being moderately knowledgeable. An inventory of 
authors or titles read pertinent to social systems 
concepts ar.d to field studies techniques yielded highly 
diversified listings. 

7. Instructional and Evaluation Materials: 

Instructional and reference materials were 
distributed both by mailings prior to the presession 
and during the presession. Some were of a general 
nature (e.g., a monograph by Richard Schmuck and 
Philip Runkel, Organizational Training for a School 
Faculty ) and others were specific to a staff presen- 
tation (e.g. i 'Organizational Structure and Teacher 
Behavior* by Philip Runkel). Several basic reference 
books were maintained on hand during the presession. 

8. Evaluation: 

Verbal feedback was elicited in informal indi- 
vidual and small group conferences during the last 
day of the presession •» written observations or cri- 
tiques were submitted by 30 percent of the participants 
following the presession. Representative observations 
were as follows with both an unenthusiastio and an 
enthusiastic comment being given: 

1. Format of the presession: 'Lack of struc- 

ture was a serious problem. We needed a tighter 
structure imposed upon us.' 'The flexibility of the 
staff enabled us to modify the presession to fit our 
highly varied wants and needs.' 

2. Content: 'Too little input from the staff » 

and too elementary in nature. We should have gotten 
more from the staff.' 'I'm going away with a differ- 
ent perspective on how to go about research. I have 
clearer ideas about critical contingencies in over- 
coming a sense of marginality in the Southern black 
student) and enough encouragement to enable me to 
proceed to intervene in a way that should be effective 
for helping change the whole system. And we should 
learn some important things in the process.' 

3. Utility/Value: 'The session was a good 

excuse for getting away from back-home pressures for 
a few days. 'I found the kind of ideas and support 
that I needed.' 

8. Director's Evaluation: 

My judgement is that the presession was of high 
value only for a few participants) of moderate value 
for the majority of participants) and disappointing to 
several participants. Participation in the session 
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did not provide much opportunity for development of 
immediate operational capabilities. However! pay- 
offs are more likely to occur over time in the design 
and implementation of field studies open to the com- 
plexities and inter-relatedness of school organizations. 

Diversity of participants' backgrounds and wants 
presented problems never satisfactorily resolved during 
the presession. Attempts to respond to the diversity 
contributed to lack of continuity and looseness in 
the presession format. In retrospect! a more effec- 
tive response would have been adherence to the initial 
schedule. In attempting to be responsive, losses 
occurred in amount of content transmittal and in dis- 
continuities in the development of related concepts 
and techniques. 

The staff was eminently qualified, including 
diverse but complementary competencies. However, lack 
of prior experience in working together (even in all 
members knowing one another) , without opportunity to 
convene prior to the evening before the presession com- 
menced, inhibited planning and preparation: uncertainty 

about instructional styles as well as critical value 
orientations prevailed until the staff actually had 
begun the presession instruction. Instructional pre- 
sentations had been carefully prepared. Further, all 
staff members were present for all sessions, and 
engaged in a continuing dialogue regarding the ideas 
under consideration. Individual and informal small 
group meetings occurred between the staff members and 
participants throughout the presession. The preses- 
sion clearly was a positive, rewarding experience for 
all members of the staff} their commitment and involve- 
ment was evident throughout the sessions. 

Social systems conceptualizations and field 
studies techniques, with reference to school organiza- 
tions in particular, are emerging as important if yet 
uncertain approaches for gaining understanding of 
educational organizations. Development of the ap- 
proaches probably would be more effective, however, 
given a more limited focus than attempted in this 
presession. 
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PRESESSION III 



1. Title: Systems Techniques in Counselor Education 

and Counseling Program Research 

2. Staff: T. A. Ryan University of Hawaii 

( Director) 

Ray E. Hosford University of California, 

Santa Barbara 



Leonard C. Silvern 

Norman R. Stewart 
Carl E. Thoresen 
Bob B. Winborn 



Education and Training 
Consultants Co., Los 
Angeles 

Michigan State University 
Stanford University 
Michigan State University 



3. General Description: 



The need for improvement and innovation in 
counseling, counselor education and related areas has 
intensified as social, political, and economic factors 
have created new problems and greater challenges for 
the educational community. With the adoption of 
standards for counselors and counselor educators the 
need for research skills was intensified. The standards 
carry an implicit mandate to the profession to make a 
searching analysis of goals and study of the ways in 
which to achieve goals most effectively. The research 
training session was designed to equip selected personnel 
with the research skills needed to implement needed in- 
novations and improvement in the field. 



Counseling, counselor education, and related 
educational programs can be conceptualized as systems. 
Therefore, it can be assumed that improvement in 
counseling, counselor education, and related educa- 
tional programs can be achieved through application of 
techniques of systems research to these areas of 
educational endeavors. 



In counseling, counselor education, and related 
areas there are needs to investigate problems and to 
arrive at best possible solutions to these problems. 

The systems research techniques of analysis, synthesis, 
modelling, and simulation can be employed to meet 
these needs. 

The acquisition of proficiency in using systems 
techniques can be accomplished in a short-term training 
session. 
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4. Objectives: 

The purpose of the presession was to develop and 
improve research competencies of counseling specialists , 
counselor educators, supervisors, educational psy- 
chologists, and researchers performing substantive 
research in counseling, counselor education or related 
areas. The program aimed to help participants acquire 
understanding of concepts, principles, and technologies 
of systems research and to become skilled in use of 
systems techniques. 

The two primary aims implementing the program 
purpose were development of participants' knowledge 
and understanding of systems research cc oepts and 
principles', and development of participants' profi- 
ciency in using systems techniques. 

Objectives implementing the presession were: 

1. For participants to demonstrate on an ob- 
jective test understanding of concepts, including 
system, feedback, analysis, synthesis, and simulation. 

2. For particpants to demonstrate on an ob- 
jective test understanding of principles, including 
feedback, closed loop, coding, lettering, signal 
paths, descriptors, F, FF, A, and error signals. 

3. For participants to demonstrate ability to 
convert a narrative problem description into a flow- 
chart model, with correct element identification and 
use of flowchart symbols. 

4. For participants to demonstrate ability to 
convert a flowchart model into narrative form. 

5. For participants to demonstrate ability to 
define behavioral objectives. 

5. Schedule: 

I^rst Day: Horning: 

Opening Pre-assessment, introductions, 
and overview. Behavioral goals, model 
for producing a system, discussion, and 
LOGOS language for flowchard modeling. 

Afternoon: 

Systems engineering of learning, systems 
including synthesis and CA1, definitions 
of system and counseling system. Sys- 
tems using feedback. 
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Second Day: Horning: 

Problem: SATELLITE! and evaluation of 

problem solutions. Model for producing 
a system model, study closed loop instruc 
tional flowchart model: Boeing. 

Afternoon : 

Analysis as a process, synthesis as a 
process. Problem guidance management. 

Third Day: Morning: 

Closed loop instructional flowchart model 
Occupational instruction and government 
based information. Problems and evalua- 
tion of solutions. 



Afternoon: 

Non-math counseling problem. Mathe- 
matical modeling: Anasynthesis , analysis 

synthesis, modeling, simulation. 

Modeling and simulation: Algebraic 

solution, probabilistic solution, and 
on-line computer simulation. 

Fourth Day: Morning: 

Evaluation of counseling problem solu- 
tions. Post-assessment. Generation of 
real life problem. 

Afternoon: 

Problems 

Fifth Day: Morning: 

Evaluation of real life problem solu- 
tions. Program evaluation. A counseling 
model . 



Afternoon: 

Model for a district testing program. 
Model for the school counselor and nego- 
tiations. Counselor education model. 



6. Participants: 

Participants came from eighteen states, the 
District of Columbia and Canada, ranged from 27 to S3 
years of age, included 40 males and five females and 
represented higher education, local school districts, 
private schools, business, industry, and government 
agencies. Out of 45 participants, 41 held the doc- 
toral degree. Distribution of participants by sex, 
age, highest educational degree, place and nature 
of employment is given in the following table. 
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Participant Characteristics 



Sex 


Number 


Education 


Number 


Age 


Years 


Male 


40 


Ph.D. 


41 


Range 


27-53 


Female 


5 


M. A. 


4 


K 


40.26 



Nature of Employment 
Employer Position 



Number 



Higher Education 

Asst. Professor 
Assoc* Professor 
Professor 
Coordinator 
Director 
Chairman 
Counseling 
Psycholog.* 8 1 

Public School 

Direotor 

Research Associate 
Supervisor 

Industry 

Director, Program 
Design 

Behavioral Scientist 
Consultant 



10 

7 

S 

3 

7 



1 



3 

1 

1 



2 

1 

1 



US 



7. Instructional and Evaluation Materials: 

The program of instruction in the presession 
assumes a prior understanding of certain basic concepts 
and principles, and ability to perform certain activities 
with ease ^ :d competence. In order to derive maximum 
benefit fi j the training program, participants mu9t have 
a thorough understanding of the language of system re- 
search, and must be able to operationalize mission goals 
and to define behavioral objectives. It is assumed 
that before the presession begins participants will be 
capable of defining problems, stating objectives in 
behavioral terms, and identifying alternatives to im- 
plement the objectives. The references listed are 
intended to provide a means by which participants can 
acquire the prerequisite knowledge and skills which are 
assumed for this program. Reference annotations are 
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provided to assist in directing reading activities so 
that optimum use can be made of participants’ reading 
time prior to the start of the presession. 

Banathy, B. Instructional systems . Palo Alto, California: 
Fearon, 1968^ X good 'overview' oT systems approach. Easy 
reading. Should be studied by everyone to insure 
thorough understanding of the nature of systems re- 
search. The appendix is particularly good. 

Boguslaw, R. The new Utopians: A study of system 

design and social change . Englewood Cliffs, n. 77: 
Prentice-Mall, 1965. An overview of systems approach. 
Intermediate reading level. 

Buckley, W. (Ed.) Modern systems research for behavioral 
scientists . Chicago"! Aldine Publishing Co., 1966. A 
collection of articles dealing with systems theory. 
Advanced reading. 

Carter, L. F. Systems approach to education: Mystique 

or reality. Educational Technology , 1969, 9, 22-31. 

Gives an overview of the systems approach, with dis- 
cussion of pros and cons from using the systems tech- 
niques . 

Churchman, C. W. The systems approach . New York: 
Delacorte Press, l£66. This brief discussion of 
systems approach gives an excellent overview of 
the total systems concept, which involves problem 
identification, objectives definition, alternatives, 
identification and evaluation. This reference helps 
to point up the way in which flowchart modeling and 
simulation techniques implement the systems concept. 

Should be studied by all participants. 

Cooley, W. W. and Hummel, R. C. Systems approaches 
in guidance. Review of Educational Research , 1969, 

39, 351-362. Relates systems techniques to guidance. 

Easy reading. 

Educational Technology , 1969, 9, No. 3, 1-77. This 
special 'issue of Educational Technology is devoted to 
counseling technology. 

Eraut, M. R. An instructional systems approach to 
course development. AV Communication Review , 1967, 

15, 92-101. Relates the techniques of systems research 
to course development. 

Gagn£, R. M. Educational objectives and human per- 
formance. In Krumholts, J. D. (Ed.)» Learning a_nd 
the educational proces s. Chicago: Rand McNally ,196$. 

pp. ”1-34. biscusses definition of objectives. Easy 
reading. 
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Mager, R. F. Preparing instructional objectives . Palo 
Alto, California: Fearon, 1962" This book tells how 

to prepare behavioral objectives. It is mandatory 
that each participant be able to define objectives in 
behavioral terms. The principles discussed in this 
reference must be thoroughly understood by each parti- 
cipant. Each participant must be able to demonstrate 
proficiency in defining objectives behaviorally . This 
can be accomplished by concentrated study of this 
reference, and practice in preparing behavioral objec- 
tives. Should be studied carefully by all participants. 

Ryan, T. A. Systems techniques for programs of coun- 
seling and counselor education. In Silvern, L. C. (Ed.) 
Applying systems engineering techniques to education 
and training. Educational Technology , 1969, 9, 1-17. 
This article describes the application of systems tech- 
niques in counseling and counselor education. It 
provides a frame of reference for the presession. Easy 
reading. Excellent bibliography on systems research. 
Should be read by all participants. The other articles 
in this issue are relevant to the presession topic. 
Casual reading is recommended. 

Silvern, L. C. Systems engineering of education I: 

The evolution of systems thinking in education . Cos 
Angeles : Education and training Consultants, 1968. 

This is the basic text for the course. Pages 111-129 
should be studied carefully by all participants. The 
• program assumes that participants will have read this 
material and have a thorough understanding of the 
concepts presented in these pages. 

Silvern. L. C. LOGOS: A system language for flowchart 

modeling. In Silvern, L. C. (Ed.), Applying systems 
engineerii*- techniques to education and training. 
Educational Technology , 1969, 9, 16-23. Contains basic 
vocabulary for flowchart modeling. Should be studied 
by all participants. 

Thoresen, C. E. The systems approach and counselor 
education: Basic features and implication. Counselor 

Education and Supervision , 1969. Discusses the applica- 
tion of systems techniques to counselor education. 

von Bertalanffy, L. Modern systems theory. New York: 
George Brasiller, Inc. T 1966. Deals with systems 
theory. Advanced reading. 

Wiener, N. Cybernetics . Cambridge, Mass.: Massa- 

chusetts Institute of Techr.olog/ Press, 1961. Deals 
with cybernetics aspect of systems research. Ad- 
vanced reading. 
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Wiener, N. Human use of human being9 , New York: 
Doubleday, 195 4. Deals with cybernetics in relation 
to systems approach to research. Advanced reading. 

8. Evaluation and Test Results: 

Two measures were taken to evaluate participant 
performance against program aims: an objective pre- 

and post-test, and self evaluation by participants. 

Evaluation of the presession in terms of par- 
ticipant achievement training objectives was accomp- 
lished by comparing pre and post instructional per- 
formances on a test intended to sample behaviors im- 
plementing program aims. The same instrument, which was 
administered for pre and post instruction testing, 
contained two subtests, both of which were intended 
to sample behaviors relating to Aim 1, understanding 
of concepts and principles of systems research. Sub- 
test 1 was concerned with basic concepts of systems 
research, excluding the concept of behavioral objectives. 
Subtest 2 was concerned only with understanding of be- 
havioral objectives. 

The median scores on both subtests for the 
posttest administration were roughly double the pre 
test median scores, as indicated by the increase from 
23 to 50 on subtest 1, and 6 to 10 on subtest 2. 

Acceptable performance criteria were defined 
for Aim 1, developing understanding of systems concepts 
and principles. The levels of an acceptable performance 
on Subtests 1 and 2 and the percent of participants 
meeting criterion levels on pre and post tests re- 
veals that 85 percent of participants reached cri- 
terion level on the post test, compared to 9 percent 
on pretest for understanding of systems concepts ex- 
clusive of behavioral objectives. On the post test 
52 percent of participants reached criterion level 
for understanding of behavioral objectives, compared 
to 24 percent on pretest. 

No immediate objective test was taken to sample 
behaviors relating to Aim 2, participant proficiency 
in applying systems techniques. A follow-up is planned 
to evaluate the progrm against this objective. 

Self-evaluations against Aims 1 and 2 were taken 
by eliciting from participants responses to indicate 
how participants felt about progress they made toward 
training objectives. 

Ninety-eight percent of participants felt the 
program resulted in their acquisition of knowledge 
abcut systems research, and 95 percent felt the program 
increased proficiency in using syftem techniques. 
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9. Director's Evaluation: 

A program evaluation was made to determine the 
extent to which program components contributed to ef- 
fectiveness of the presession. Data were gathered to 
evaluate learning activities, instructional materials, 
program content, and program organization. 

Participants rated program learning activities 
on a four-point scale to indicate degree to which the 
activity contributed to achievement of program objec- 
tives. Mean ratings reveal that all activities were 
rated above the mean expected by chance. The activity 
deemed most worthwhile was the audio-visual presenta- 
tion, with lectures and work on real life problems 
rated next in importance. General discussion was 
rated lowest < 

Evaluation of instructional materials was made 
by participant rating on a four-point scale of six 
references which were required for the program. Mean 
ratings reveal that all references were rated above 
the chance mean. The reference rated as most worth- 
while was LOGOS? A system language for flowchart 
modeling , by L. C. Silvern, with the next highest 
rating for Systems techniques for programs of coun - 
seling and counselor education by T. A. Ryan. 

Program content was evaluated by participant 
rating on a four-point scale of each program unit in 
terms of contribution to program aims. Mean ratings 
reveal that units considered most valuable were 
Problem from Real-Life Environment, Conceptual Analysis 
and Synthesis i and Rules and Symbols for flo wchart* " 
Modeling" TKe unit rated lowest was the Counselor 
Education Problem . All units were rated above the 
chance mean. 

Program Management was evaluated by participant 
ratings of aspects of program organization and manage- 
ment relating to information, meals and lodging, staff 
qualifications, tine utilization, and climate for 
learning. Participant ratings of program management 
indicate general satisfaction with program information, 
meals and lodging, staff competencies, and climate for 
learning. The physical facilities were not considered 
satisfactory. There was no consensus with regard to 
time, with one-fifth to one-fourth of the participants 
expressing the opinion that time wa9 not sufficient. 
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Research and the Development of Instruc- 
tional Theory 



PRESESSION IV 

1. Title: 

2. Staff: Thomas J. Shuell 

(Director) 

Joe L. Byers 
Frank H. F.»rley 
S. Jay Samuels 
Fred Linder 

3. General Description: 



State University of New 
York at Buffalo 

Michigan State University 

University of Wisconsin 

University of Minnesota 

State University of New 
York at Buffalo 



The main emphasis of this presession was direc- 
ted toward the acquisition of the knowledge and skills 
necessary for conducting research on the psychology 
of instruction and for developing a viable theory of 
instruction. Particular emphasis was placed on the 
ability to ask relevant questions concerning the pro- 
cess of learning and instruction. Various theoretical 
approaches to the study of instruction were considered, 
and selected research on human learning and instruction 
was presented and discussed. An attempt was made to 
integrate present-day knowledge and to outline some 
directions which future research on the psychology of 
instruction might take. 



Each participant was asked to state a poten- 
tially researchable question dealing with the psychology 
of instruction. These questions were th< n discussed 
critically by the staff and the other participants in 
small group discussion, and each participant then de- 
veloped an experimental study that would help answer 
his question. 

While much of the instruction of the presession 
was lecture/discussion, ample time was provided for 
the participants to interact with the staff and with 
one another through daily discussion periods. 

4. Objectives: 

The primary objective of the presession was to 
encourage research and theory development in the psy- 
chology of instruction by providing the participants 
with the necessary training for pursuing such en- 
deavors. More specifically, the things which it was 
hoped that the participants would be able to do at the 
completion of the presession included the following: 
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1. Ask amenable questions pertaining to the 
psychology of instruction. 

2 . Outline a study which would provide an 
adequate test of one of the above questions. 

3. Identify relevant and irrelevant variables 
in the investigation of the psychology of instruction. 

4. Differentiate between learning and instruc- 
tion. 

5. Distinguish between the psychology and the 
philosophy of instruction. 

6. Distinguish between questions concerning 
what is unknown about the process of learning and/or 
instruction and the application of present-day knowledge 
to the solution of practical problems. 

7. Differentiate between research and devel- 
opment and describe how the two interact. 

8. Describe the main characteristics of various 
approaches, i.e.» theories, models, etc., to the study 
of instruction. 

9. Describe some of the ways in which instruction 
can be adapted to meet various individual differences. 

10. Differentiate between cognitive and affec- 
tive factors in learning and instruction. 

5. Schedule: 

First Day: Morning: 

Discussion of presession objectives. 

The psychology of learning and the psy- 
chology of instruction. Some theories 
and models of the instructional process. 

Afternoon: 

Research and development in education. 

Review of some aspects of scientific re- 
search and the role of theory and/or models 
including a consideration of the inter- 
action between theory, data, and method- 
ology. The asking of relevant questions. 
Panel discussion by staff of day's pre- 
sentation beginning with short summary. 
During the discussion it is hoped that 
each staff member's orientation will be- 
come apparent. 
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Second Day: 



Third Day: 



Fourth Day: 



Fifth Day: 



0 




Morning: 

Some methodological considerations in 
instructional research . Attention in 
learning. Attention and its role in in- 
struction. Motivational and affective 
factors in learning and instruction: 

Some issues in current research. 

Afternoon: 

Motivation and attention: A consideration 

of methodology, technology, etc. , and some 
directions for the future research. In- 
dividual differences in learning and in- 
struction. Small group discussions cen- 
tered around interests and problems of 
participants. Individual consultations 
with staff members. 

Morning: 

Individual differences by instruction in- 
teraction: Some current research. The 

process of reading acquisition: Learning 

models and some current research. 

Afternoon : 

Behavior management and its implication 
for instructional theory. Critique of 
participants' questions in small group 
discussions . 

Morning: 

The process of reading acquisition: Im- 

plications for instructional theory and 
areas of needed research. Learning from 
written materials: Current issues and 

research. 

Afternoon : 

Learning from written materials: Pro- 

grammed instruction, and implications for 
future research. Individual consultations 
with staff. Critique of participants' 
questions and proposed research in small 
group discussions. 

Morning: 

Concept learning and instruction: Some 

implications for instructional theory 
and future research. Small group dis- 
cussions . 

Afternoon: 

Critique and discussion of participants' 
questions and research proposals — small 
group discussions and/or individual con- 
sultation with staff. 
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6. Participants: 

Thirty participants from the United Staten and 
Canada attended the presession. Nineteen (63 percent) 
of the participants hold academic appointments in 
universities and colleges, and 4 (13 percent) of the 
participants are employed by either a regional labora- 
tory or an R 6 D center. Fourteen (47 percent) of 
the participants had attended a previous presession. 
Doctoral degrees were held by 25 (83 percent) of the 
participants. The participants had published an 
average of 3.57 research articles in scholarly journals. 
Sixteen (53 percent) of the participants indicated 
that they had been a director or co-director of air 
least one research project funded by a granting agency 
such as USOE, NIMH, etc. 

7. Instructional Material: 



Handout materials were prepared by the staff and 
used during the presession. 

8. Evaluation: 



A questionnaire designed for this presession only 
was mailed to the 30 participants approximately two 
weeks after the end of the presession. The question- 
naire consisted of 10 rating-type items and four open- 
response items. Response to the questionnaire was 
anonymous. Twenty-two individuals returned the ques- 
tionnaire, and the results for these respondents are 
presented below: 

1. Do you feel that the objectives of the presession 
were met? 



Definitely (32%), Somewhat (50%), No (9%) Undecided (9%) 



2 . 

met? 



Were your objectives for coming to the presession 



Definitely (36%) Somewhat (41%) No (18%) Undecided (5%) 

3. Do you feel that you will use what you have learned 
in this presession in your future work? 

Definitely (68%) Probably (23) No (5) Undecided (4%) 

4. Was the presession staff sufficiently available 
for consultation on individual questions and problems? 

Definitely (82% )Somewhat (14%) No (4%) Undecided (0) 
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5. Was there sufficient opportunity for interaction 
with the staff and other participants? 

Definitely (68%) Somewhat (18%) No (5%) Undecided (5%) 

6. Was the pacing of the material: 

About right (77%)Too Fast (5%) Too Slow (9%) 

Undecided (9%) 

7. Was the level of difficulty of the material: 

About Right (59%) Too Difficult (0) Too Easy (27%) 
Undecided (9%) 

8. Was the coverage of present research and theory 
adequate? 

Definitely (32%) Somewhat (36%) No (18%) Undecided (14%) 

9. Were the implications for future research suf- 
ficiently emphasized? 

Definitely (41%) Somewhat (27%) No (27%) Undecided (0) 

10. Do you feel that this presession should be offered 
again? 

Definitely (77%) possibly (14%) No (5%) Undecided (5%) 

The questionnaire included open-ended items 
asking the participants for specific comments and 
suggestions. However, it is somewhat difficult to 
summarize these comments in any systematic or mean- 
ingful way. In many respects the comments seem to deal 
with individual preferences and interests. The diversity 
of interests, expectations, and preferred mode of in- 
struction among the participants is quite clear from 
the comments. For example, the attempt to get each 
participant to ask a relevant and researchable question 
concerning the psychology of instruction and to develop 
a research design for testing the question drew the 
following two comments: 'More time on staff presenta- 

tions, less time on development of research projects. 
Project, was a kid-stuff assignment.' 'The small group 
discussions of individual members' research proposals 
were excellent. To be repeated in the future, by all 
means ! ' 



In responding to an item on the special ques- 
tionnaire which asked those who felt that their ob- 
jectives for the presession had not been satisfied to 
list their objectives which they felt were not ade- 
quately met, five individuals indicated that they had 
wanted more emphasis on the nature of theory and various 
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aspects of theory development. A number of individuals 
said that they thought the presession could have been 
better organized. 

In addition, several individuals made sugges- 
tions which I feel are particularly useful ways in which 
the presession could have been improved. One person 
suggested that the small group discussions center around 
the interests of the participants rather than the in- 
terests and competencies of the staff. While no sugges- 
tion was made as to how this could be accomplished within 
the limitations of staff availability, it is possible 
that a staff member might not be required in each group. 
Another ’ndividual suggested that staff consultation 
time and t>tudy time be placed in the middle of the 
daily schedule so that the individuals would not be 
too tired or drift away as was the tendency when this 
activity was scheduled at the end of the day. Whether 
or not this would help solve the problem in any real 
sense is not clear, since it is possible that they 
would leave anyway and miss the last regular session 
of the day. A third individual suggested that more 
time should be spent discussing preassigned readings. 
These could be read before the presession or time could 
be allowed at the possession for reading Them. Fin- 
ally, one individual suggested that a glossary of 
terms be compiled to facilitate learning and discussion 
for those unfamiliar with some of the terms used. 

9. Director's Evaluation: 

In general, both the staff and the participants 
felt that the presession was successful. Probably the 
best single index of this success is the large marjority 
of participants who indicated on the questionnaires 
that the presession should be offered again. 

The backgrounds, interests, and expectations of 
the participants were quite diverse, and better ways 
to individualize instruction could probably have been 
found. If we had had a better indication of these, 
differences we could probably have done a better job 
of taking them into account. The information requested 
on the application form is of little usefulness in pro- 
viding the type of information needed. Perhaps some 
form of pretesting, either before the presession be- 
gins or on the first day of the presession, would be 
useful. As it was, the attempts which the staff did 
make to individualize instruction and to provide some 
flexibility in the presession were often interpreted 
by the participants as reflecting a lack of structure 
and organization. It might prove worthwhile to have 
individuals with different backgrounds and interests 
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working in different subgroups. This way, it might 
be possible to carry on instruction at several levels 
at the same time. 

One shortcoming of the presession was the fail- 
ure to adequately convey the various objectives of the 
presession to the participants. Although attempts to 
articulate the objectives were made by the staff on the 
first day and throughout the presession, it is clear 
from some of the comments that we were not always suc- 
cessful in this regard. It appears to be important 
for the objectives to be stated in writing and made 
available to the participants. It would probably be 
useful to send them to the participants prior to the 
presession. Also, several participants indicated that 
they would have appreciated receiving a reading list or 
the bibliography prior to the presession. 

It would be very desirable for the staff of a 
presession being offered the first time to be able to 
meet together several times for planning sessions. It 
is sometimes extremely difficult to do effective plan- 
ning over the phone and via the mail. This is especially 
true if the staff members have not worked together in 
some way before. Although I was able to meet with each 
staff member for approximately a half day during the 
early stages of planning the presession, it would have 
been very useful if we had been able to meet as a group, 
perhaps during the middle or latter stages of planning. 

A fairly extensive library of books and articles 
relevant to the topic of the presession was provided. 

One or two copies of almost every article on the bibli- 
ography was available for the use of the participants. 

The participants felt that this library was extremely 
useful . 



Probably the most important function of the pre- 
session should be evaluated by the extent to which the 
presession encourages future research on the psychology 
of instruction. This aspect, of course, is extremely 
difficult to evaluate at the present time. 




PRESESSION V 






1. Title: Applied Linear Regression Analysis in 

Educational Research 

2. Staff: Joe H. Ward, Jr. 

(Director) 



Earl Jennings 
Robert Bottenberg 



Janos Koplyay 



3. General Description: 

The present presession, Applied Linear Regres - 
sion in Educational Research w as conducted by the same 
personnel and had the same orientation as the previous 
three presessions which held the name Multivariate 
Design and Analysis in EducationalResearch . The name 
change is in itself significant, first, the new name 
is much more in keeping with the content of the pre- 
session. In the past, participants had occasionally 
come to the presession with the idea that the content 
would be oriented toward such topics as multivariate 
analysis of variance, discriminant analysis, canonical 
analysis, factor analysis, and related multivariate 
techniques and might rightfully feel that the title of 
the presession was misleading. Another significant 
point can be made as regards the name change, however. 

In each past evaluation of the presession, participants 
have been allowed to answer an open-ended question re- 
garding suggestions for future presessions. One of the 
suggestions made was to change the name to include the 
multiple regression approach, rather than use the term 
multivariate. The point being made here is that this 
is only one indication as to how the presession staff 
has favorably reacted to criticism; this ongoing evalu- 
ation has contributed to a continually improving mode 
of presentation. 

4. Objectives: 

The session is designed for education researchers 
who have the basic statistical tools in their reper- 
toire, but because of the rapid improvement of computer 
techniques for the systematic organization and analysis 
of data are presently unable to formulate research 
problems for computer analyses that will yield answers 



Southwest Educational 
Development Laboratory, 
Sen Antonio, Texas 

University of Texas 

Air Force Personnel Re- 
search Division, AFHRL, 
Lackland AFB, Texas 

Air Force Personnel Re- 
search Division, AFHRL, 
Lackland AFB, Texas 
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to the questions at issue. In general, the partici- 
pant received his statistical training in the more 
traditional analytical procedures, and is unable to 
formulate his own problems and devise the most mean- 
ingful analysis. 

The primary objective of this session is to 
assist the participants in developing techniques of 
formulating research problems for computer analyses so 
as to make full use of the multiple linear regression 
approach. Specifically, the participants will be able 
to : 



1. Define vectors that express his conceptuali- 
zation of a problem. 

2. Formulate models appropriate for his specific 
problems without conforming to experimental iesigns for 
which prescribed computational procedures are available. 

3. Identify vectors that represent information 
that has been measured on a continuum. 

4. Define vectors so as to express nonlinear 
and interaction relationships. 

5. Use categorical and continuous vectors in 
models developed to remove the 'contamination' of 
other factors (logic of covariance analysis). 

6. Apply an unambiguous set of rules to the 
determination of the appropriate degrees of freedom to 
be used with the linear regression model. 

7. Cite novel examples of research problems to 
which linear regression is applicable. 

This session is designed to develop the apprecia- 
tion of multiple linear regression as a general approach 
to the formulation and analysis of research problems. 

As such, the activities will be divided about evenly 
between lecture-discussion, laboratory exercises re- 
lated to the objectives listed and exercises related 
to appropriate computer operations. The illustrative 
problems will come largely from the behavioral sciences. 

Concepts and exercises will be introduced 
systematically as they are required in the logical 
development of the materials. Participants will have 
direct experience with data processing and computer 
equipment. Each participant will prepare a problem 
statement which reflects acquisition of concepts and 
development of the attendant techniques that are use- 
ful in conceptualizing research problems. 




65 . 



5. Schedule: 



First Day: Morning: 

Overview, objectives. Evaluation of Pre- 
attendance. Vector concepts. One attri- 
bute mutually exclusive categories. Three 
services problem. Linear dependence and 
partitioning a set of vectors into inde- 
pendent and redundant sub-sets. Funda- 
mental vector operations 

Afternoon: 

Comparison of assumed and restricted 
models. Computation of the F statistic. 
Three services computer output. Equiva- 
lent models. One attribute problems in- 
volving mutually exclusive ordered cate- 
gories and linearity. Application of the 
one attribute analysis. 

Second Day: One-attribute computer output. Extensions 

of one-atribute analysis. Fifty stares, 
stanines, twc-lines, several lines, second 
degree polynomial. Extensions of one attri? 
bute analysis. Imposing restrictions to 
test for second degree polynomial. 

Afternoon: 

Logic of computer program. Assumptions of 
general linear model. Details of computer 
program, card preparation, DATRAN. 

Third Day: Morning: 

Two-attribute problems. Equivalent model 
for two-attribute interaction. Main 
effects . 



Afternoon: 

Two-attribute computer output. Extensions 
of two attribute analysis. Paired obser- 
vations t-test, treatment by subjects, 
missing cells, extreme missing cells. 
Two-attributes with one ordered. Repara - 
metrization. Iterative method of computing, 
PERSUB. 

Fourth Day: Morning: 

Three-group extension. Three-group 
analysis computer output. Extensions of 
two-attributes one ordered. Intersec- 
tion at a specific point, main effects, 
polynomials. Extensions of two-attribute 
one-ordered. Interaction test, one-attribute 
with polynomials ordered. 
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Afternoon: 

Two-attributes both ordered. Two attri- 
butes both ordered computer output. Exten- 
sion of two attributes both ordered. Non- 
linearity, complex interaction. Inter- 
action test, both attributes ordered with 
polynomials. Orthogonal decomposition, 
and least squares. 

Fifth Day: Horning: 

Two attributes, controllable attribute 
ordered (VARICO). Synthesizing regression 
models. Further extensions. Discontinuity, 
change detection, attributes influenced 
by treatments. 

Afternoon: 

Regression models in judgement analysis 
and hierarchical grouping applications. 
Further extensions, summary, evaluation. 

At various times during each day, laboratory sessions 
were scheduled. 

6. Participants: 

Of the 4? participants, Ml weie male. Their mean 
age was 34.3. On the average, they have produced 2.2 
research articles and spend about 5? percent of the 
time in research. Twenty-threo possessed a doctorate. 
Fifteen have received funds to conduct research. 

Their distribution around the country is as follows: 

East Coast 14 Canada S 

West Coast 2 Hidwest 23 

Mountain States 3 South 3 

7. Instructional and Evaluation Materials: 

For the past two years, the text by Bottenberg 
and Ward, Applied Multiple Linear Regression , has been 
mailed to participants after the staff received a 
notification of acceptance from the participants. They 
were also sent a copy of Activity 1. On the first day 
of attendance, the participants were given a closed 
book evaluation, which was another copy of Activity 1. 
Each participant was also asked two questions: 

Did you receive the book? 

Did you read the assignment? 

Activity 1 was graded by the staff, with the 
following results: 
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Of those who had read the assignment, 35 did 
reasonably well on Activity 1, 4 did not do particularly 
well on Activity 1. 

Of those who had not read the assigned reading, 

1 did reasonably well on Activity 1, 3 did not do par- 
ticularly well on Activity 1. 

These results aru considerably better- than was 
true with the pre-mailing the previous year. With the 
rather inefficient job that the AERA has done in the 
distribution of the Educational Researcher (which 
contained the application blank) , these results can be 
seen to be even more gratifying to the presession staff. 

8. Evaluation and Test Results: 

Immediately after the presession had been com- 
pleted, the participants filled out an evaluation form. 

The questions are repeated here with totals regarding 
each question. Also included are comments felt worth 
remembering by the editors. It is important to note 
items 1-a through 24 were open-ended. The tallying 
done here is an interpretation of the participants open- 
ended response. 

Directions : Please respond with a word, a phrase, or 

one or more sentences to as many of the following ques- 
tions as you can. Your frank and honest evaluation can 
only benefit everyone concerned. Do not identify 
yourself by name unless you prefer to do so. 

Environment and Facilities 

la. To what extent did the relative availabiltiy or 
unavailability of books and journals interfere with or 
promote your attempts to master the content of this 
session? 

Favorable response (28) Unfavorable response (4) 

Neutral response (8) Did not respond (3; 

lb. To what extent did reproduced materials given to 
you by the staff improve matters? 

Favorable response (38) Unfavorable response (0) 

Neutral response (2) Did not respond (3) 

"Xerox copies savsd a lot of time. The 3-ringed note- 
book and the 'butterscotch* book made this presession. " 

2a. *>id you feel that you lacked a 'place to work,' either 
alone or in small groups? 




Yes (2) No (37) Neutral (4) 

2b. Wa9 your hotel room satisfactory? 
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Yes (33) No (2) Neutral (8) 

3a. Which features of the meeting rooms were inadequate 
or not conducive to learning? 

"Coffee was expensive." "Not enough elbow room." 

3b. Which features were especially facilitative in the 
same regard? 

"Some refreshments always available." "Plenty of room 
at the tables to spread out." "The arrangement of 
refreshments. " 

Scheduling and Organization 

4a. Was five days too long a period to leave your 
work at home for the purpose of attending this session? 

Yes (6) No (36) No response (1) 

4b. Was five days too short a period in which to learn 
much of the content of this session? 

Yes (20) No (19) Omit (4) 

"Probably so. The basic approach can be understood in 
this time i but many of the tricky applications require 
more time." " In some respects, yes--I suspect that a 
number of participants wi-1 suffer from isolation star- 
vation as they return to their home environments." "I 
believe that I could profit from either additional days 
or another session to try and digest the many models 
and other excellent idoas presented." 

Sa. Were you allowed enough time in which to pursue 
activities of your own choosing? 

Yes (24) No (16) Omit (3) 

Sb. Would you have preferred not to meet in the 
evening after dinner? 

Yes (26) No (16) Omit (2) 

"No, but this implies : ... <:.ys and/or sessions." 
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5c. Would you have preferred more or fewer meetings 
per day than there actually were? Or was the number 
of meetings per day agreeable to you? 



Need more (0) Agreeable (30) Need less (6) Omit (5) 



"Day meetings agreeable but should omit night meetings." 
"I think number of meetings ip. OK, but it should leave 
some time for digesting." "Agreeable (Except I prefer 
nights for own study)." 



6a. Were the individual lectures too long to sit and 
listen or take notes? 



Yes (14) No (27) Omit (2) 



"Pacing was good--lecturers broke when there was a need." 

6b. Were the lectures scheduled in an appropriate 
sequence? 

Yes (35) No (2) Not sure (6) 



7. Did you have sufficient time to interact with other 
participants? 

Yes (28) No (12) Cmit (3) "No--for some reason we 
worked so hard that we didn’t get together much." 

"No, I would have preferred at least one short meeting 
to find out what applications others were making or 
anticipating making of materials." 

8. Were the instructors too inaccessible or unapproach- 
able so that you did not get the individual attention 
that you desired? 

Yes (2) No (39) Occasionally (2) 

"They were readily available from 8 a.m. to 10 p.m.... 
very willing to assist. " "Instructors were very ac- 
cessible and very approachable." 

9. Did the attempts to evaluate your progress and 
reactions during the session (and at this moment) 
interfere with your work here? 

Yes (0) No (38) Not sure (5) 



10. In general, was the presession well organized? 
Yes (41) No (0) Not sure (2) 
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Content and Presentation 

11a. Did the content of the lecturos and readings pre- 
suppose far more training than you had? 

Yes (10) No (26) Occasionally (7) 

"No, but the lecturers went too fast." "Give one 
hour summary on the first day." "Sometimes, but not 
often." "To presuppose less would, have made the sessions 
worthless. " 

lib. Should less training in these areas or more have 
been presupposed? 

Less (12) More (8) Same (20) Omit (3) 

12. To what extent was the content of the lectures 
and readings relevant to what you hoped to accomplish 
during the session? 

Favorable response (31) Unfavorable response (0) 

Neutral (7) Omit (5) 

"More on direct applications--less on replacement of 
other statistical applications . " "Maybe fewer models 
with more depth in selected ones." 

13a. Were the lectures stimulating and interesting? 

Yes (3S) No (0) Ujuully (8) 

13b. Were the lecturers competent to spoak on the 
subjact assigned them? 

Yes (42) No (0) Usually (1) 

13c. Were the lecturers well prepared? 

Yes (MO) No (1) Usually (?) 

1M. Were you disapponted in any way with the group of 
participants? 

Yes (1> No (M2) 

Answer each of the following only by checking the 
more appropriate blanks 

IS. If you had it to do over again * would you apply 
for this presession which you have just completed? 




Yea (M2) No (1) 
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16. If a presession such as this is held again, would 
you recommend to others like you that they attend? 

Yes (43) No (0) 

17. Do you anticipate maintaining some sort of contact 
with at least one of the presession staff? 

Yes (33) No (10) 

18. Do you feel that AERA is making an important con- 
tribution to education by sponsoring presessions such 
as this one? 

Yes (43) No (0) 

19. Do you feel that anything has happened during these 
five days to make it more likely that you will leave 
your present position of employment? 

Yes (3) No (40) 

20. I 8 it likely that you will collaborate in research 
with someone else attending this presession (other than 
those you already were likely to collaborate with)? 

Yes (9) No (33) Unsure (1) 

21. Do you feel that the staff should feel that it has 
accomplished its objectives during this five-day pre- 
session? 

Yes (43) No (0) 

The participants were asked to make comments 
and suggestions concerning the presession. Their sug- 
gestions, and the number of times each suggestion was 
made, can be summarized as follows: 

1. An advanced session to follow up this pre- 
session should be arranged. (6) 

2. Participants need more statistical back- 
ground. (3) 



3. The entire book ( Applied Multiple Linea r 
Regression ) should be read before coming .TD * 

4. More geometric (graphical) interpretations (1) 

5. Cut some examples--concentrate on main 
ones. (1) 

6. Break up lectures with more activities (1). 
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7. More time on complex concepts. (1) 

8. A bibliography of work in this field would 
be helpful. (1) 

9. Allow time for individual problems. (1) 

9. Director's Evaluation: 

It is worthwhile to compare this list of sugges- 
tions with a similar list from the previous years' 
evaluation. In 1988, only three suggestions were 
given more than three times: 

1. Send out materials (such as Bottenberg and 
Ward's textbook) before the presession. 

2. An advanced session to follow up this pre- 
session should be arranged. 

3. More attention should be given to individual 
problems . 

The first of these suggestions was implemented 
for the 1969 presession; the second and third sugges- 
tions in 1968 became the first and second most often 
given in 1969. Implementing the suggestion of having 
an advanced follow-up session continues to be the 
leading suggestion, and its implementation would seem 
to be able to draw a competent and enthusiastic audience. 
Specifically, implementing an advanced follow-up 
session might well necessitate federal funding, if it 
were to be adequato both in scope and being given to the 
intended audience. An application for such a post- 
session is now under consideration (tentatively sched- 
uled for the summer of 1970). Hopefully, funding can 
be arranged. 

The second suggestion concerning allowing time 
for individual problems was specifically mentioned 
only once. This can be interpreted to mean that the 
staff is trying to extend help in this direction. 

Other suggestions given in 1969 included: 
number the handout sheets, material on overhead pro- 
jector could be given in handouts $ and title of pre- 
session should refer to the multiple regression ap- 
proach. The staff has implemented all three of these 
suggestions. The first two suggestions were combined 
into the production’ of what might be considered as 
a detailed chronology of the presession. The change 
in name of the presession quite obviously has also teen 
effected. 

The 1970 suggestions tend to be more oriented 
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to specifics of presentation rather than more concrete 
suggestions for change in direction. Several of the 
suggestions are not really within the presession staff's 
control. Apparently, the presession as an entity has 
reached full maturity. 

Some comments (open-ended)by the participants 

follow: 



"The presession acceptance should instruct the 
people admitted to read the entire text carefully, 
rather than 30 pages. Perhaps a pre-quiz which en- 
compassed the entire book would be advantageous." 

"The only point which I feel could be expanded 
is to the introduction of the session and the preamble 
to the individual activities. I felt I was not as 
ready to define the specifics that were involved in 
each problem." 

"We desperately need (and also want) to push 
this further. I'd like a presession for past parti- 
cipants to push things further," 

"For the relatively novice statistician, this is 
just too much to understand all at once. The first 
two days seemed very manageable, but my ability to 
appreciate some of the points made from then on 
diminished rapidly." 

"I never really had a traditional regression 
course, as such. This may be a help or hindrance, I 
don't know. What does concern me is my ability to 
communicate what I have learned about a 'new' research 
tool to people far more powerful th*\n me in use of 
traditional statistical methods. It seems to me. 

I almost have to be at the same level of sophistica- 
tion in order to promote acceptance." 

"The content of this session should form a 
course (one semester at least) of all psychology and 
educational psychology departments of all the nation's 
graduate schools." 

"Excellent presentation, enjoyable." 

"Learning and mastery should be greatly facili- 
tated by the comprehensive materials distributed at 
this presession. Staff and participants were all most 
helpful and enthusiastic. Thank you very, very much." 

"The presession should be at least two weeks 
long (although this might imply changing the name to 
'summer post-sessions' or 'summer institutes'). If of- 
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fered during summer, a full summer session sounds much 
better. " 

"All in all, I thought it was great." 
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PRESESSION VI 

1. Title: Human Behavior Genetics and Its Implica- 

tions for Educational and Behavioral Re- 
search 

2. Staff: Steven G. Vandenberg 

(Direct'' >) 

Sartdra ocarr 
Arthur R. Jensen 



Kenneth R. Henry 

3. General Description: 

Part of the session was aimed at a broad audience 
of those who wish to learn some of the practical impli- 
cations of recent advances without getting involved 
with too many technical details. Administrators con- 
cerned with policy would thus be able to base their 
decisions on informed opinion. At the same time, most 
of the lectures were at a somewhat more technical 
level and included reference to appropriate statistical 
or laboratory techniques. This part of the presenta- 
tion was aimed at researchers, research advisors, and 
those teaching any courses which may touch on hereditary 
contributions to behavior. 

4. Objectives: 

The aim of the presession was to bring the 
participants up to date on the recent advances in human 
genetics, many of which are relevant to behavior. These 
advances have made more understandable the role of genes 
in regulating behavior, and in several instances allow 
prevention or cure of the behavioral defects. In turn, 
the possibility of treatment has changed the attitudes 
toward genetics. Instead of fatalistic acceptance of 
hereditary defects, there now is the hope of discovering 
ways to intervene in ever growing numbers of cases. Dis- 
cussion of the discovery of each oytologloal or bio- 
chemical abnormality was to be related to the behavioral 
consequences. After the single gene substitutions and 
chromosomal abnormalities, discussion shifted to quan- 
titative genet ics, using height, intelligence and 
personality as the examples. Data from several types 
of 'natural experiments' were reviewed such as twin 
studies, adoptive children, inbreeding, and incest. 

Several statistical approaches to estimating herltabilities 
were contrasted. In other lectures the acquired in- 
sights were utilised in approaching such applied problems 
as: Is the national IQ declining? Are there racial 

differences in IQ? Why do deleterious 'raits continue 



University of Colorado 

University of Pennsylvania 

University of California, 
Berkeley 

University of Wisconsin 
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to exist even when they lower genetic fitness? 

5. Schedule: 

First Day: Morning: 

Movie, 'How Life Bogins.' Review of 
Mendel *8 laws. Examples in man of domi- 
nant and recessive inheritance. Sex 
determination, anomalous color vision. 

The story of PKU, discovery, biochemical 
cause, diagnosis and mass screening, 
treatment by diet, carrier detection, 
genetic counseling, amniocentesis and 
therapeutic abortion, possibility of 
'cure' by implantation of enzyme capsule. 
Other examples of inborn errors of metab- 
olism (galactosemia, Gaucher's disease). 

Afternoon: 

Lecture by Dr. V. Elving Anderson on 
Genetics of Mental Retardation. Film, 
'Medical Genetics, Part 1.' Mitosis and 
Meicsis. Non-disjunction causing aneuplo- 
idies such as Down's syndrome, Turner's 
and Klinefelter' 8 . Crossing over, linkage, 
mapping of chromosomes. Sexlinkage. 
Linkages established in man. 

Evening: 

No host cocktail party for participants 
and staff of all workshops at the Leaming- 
ton Hotel: AERA Central Office Staff will 

be present. 

Second Day: Morning: 

Film, 'Medical Genetics, Parts II and III.' 
The genetic code, theory of evolution il- 
lustrated by haemoglobins in man and other 
species. Balanced polymorphism illus- 
trated by sickling 6 malaria. 

Afternoon: 

Effect of sex chromosome aneuploidies on 
intelligence; a*e XYY predisposed to 
crime? Quantitative genetics. Approxi- 
mation to normal distribution with rela- 
tively few genes. Definitions of herita- 
bility. Additive variance, variance due 
to epistasis. Heritability from twin 
studies; from parent-offspring data. As- 
sumptions of twin studies. Studies of 
adopted children. 
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Third Day: 



Fourth Day: 



Fifth Day: 
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Evening: 

Evidence for quantitative inheritance of 
intelligence from inbreeding and incest. 
Assortative mating) mate selection and 
their genetic consequences. 

Morning: 

Findings of twin studies with respect 
to intelligence) special abilities) per- 
sonality, crime, homosexuality. Evidence 
from twins raised apart. 

Afternoon : 

Multivariate analysis of twin differences. 
Cattell: MAVA Method, Elston's proposals. 

Conceptual relationship between instincts 
and heredity. Imprinting, critical periods. 
Evblution of the mammalian nervous system. 

Evening: 

Discussion group. 

Morning: 

Evolution of sensory systems. Evolution 
of learning and problem solving. Hodos 
criticism, evolution of human society. 
Possibility of directed evolution. H. J. 
Muller' - proposals. Positive and nega- 
tive eugenics. Cattell 6 Horn's theory 
of fluid and crystallized intelligence. 

Afternoon: 

Animal behavior genetics: special methods 
such as selective breeding, use of iso- 
genic strains, cross-fostering, diallel 
crosses and other designs. Genetics of 
audiogenic seizures. 

Evening: 

Free 

Morning: 

Is the national IQ declining? The money 
value of an IQ as estimated from expected 
life earnings. Social mobility. Are 
<te headed toward a meritocracy and class 
society? Moderating influence of educa» 
tion (nursery, elementary and secondary 
school and college). Are there racial 
differences in IQ? Arguments for and 
against. Evidence from cross cultural 
studies. How could this question be an- 
swered. 
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Afternoon : 

Future directions of research in human 
behavior genetics. Lecture by Dr. Irving 
I. Gottsman on genetic factors in schizo- 
phrenics. Administration of evaluation 
material. Short achievement test. 

6. Participants: 

Thirty-four educators and researchers attended. 

7. Instructional and Evaluation Materials: 

The lectures were evenly distributed between 
the staff, so that Jensen, Scarr and Vandenberg lec- 
tured about one-third of the time. Dr. Scarr was 
replaced on Sunday and Monday by Dr. Henry. The actual 
topics discussed are shown in the schedule above. 

Shortly before the first lecture, a list was distributed 
of 64 technical terms commonly used in Behavior Genetics 

The following reprints were also distributed: 

By S. G. Vandenberg: 

Contributions of Twin Research to Psychology. 
Psychol. Bull . 1966, 6: 327-352. 

Hereditary factors in psychological variables 
in man, with a special emphasis on cognition. 

In James S. Spuhler (Ed.) Genetic diversity and 
human behavior , Chicago, Aldine, 1967. 

The Nature and Nurture of Intelligence. In 
David C. Glass (Ed.) Biology and Behavior: 
Genetics , New York, Rockefeller University 
Press and Russell Sage Foundation, 1968. 

Humar. behavior genetics: present status and 

suggestions for future research. Merrill Palmer 
Quarterly of Behavior and Development, 1969, “ 

15: 121-154. 

From the Life Educational Reprints Series: 

Life Before Birth 

DNA - The Secret of Life 

From W. H. Freeman (Scientific American Reprints) 

W. A. H. Rushton - Visual Pigments in Man 
Marshall W. Nirenberg - The Genetic Code: II 

Edward F. MacNichol, Jr. - Three Pigment Color 
V ision 

Emile Zuckerkandl - The Evolution of Hemoglobin 
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F. H. C. Crick - The Genetic Code: III 

David C. Phillips - The Three Dimensional Struc- 
ture of an Enzyme Molecule 

Anthony C. Alli3on - Sickle Cells and Evolution 

Ruth Hubbard 6 Allen Kropf - Molecular Isomers 
in Vision 

C. A. Clarke - The Prevention of 'Rhesus' Babies 

Luigi Luca Cavalli-Sforza - 'Genetic Drift' in 
an Italian Population 

The evaluation of the degree to which the objec- 
tives of the presession workshop on Behavior Genetics 
were achieved is based on three sources of information: 

1. A subject mastery test. 

2. A questionnaire. 

3. Comments in letters. 

8. Evaluation and Test Results: 

The results of the test of subject matter admin- 
istered showed a gratifying performance. The percentage 
of persons giving the correct answers to each of 24 
items is shown below. The test was administered in two 
parts to interfere less with the lectures. 



Item 


% Correct 


Item 


% Correct 


Item 


% Correct 


A-l 


95.5 


A- 9 


4.5 


B-7 


77.0 


A- 2 


81.8 


A-10 


86.4 


B-8 


77.0 


A-3 


81.8 


B-l 


69.2 


B-9 


53.8 


A- 4 


54.5 


B-2 


100.0 


B-10 


15.4 


A-5 


59.1 


B-3 


30.8 


B-ll 


84.6 


A- 6 


77 . 3 


B-4 


69.2 


B-12 


53.8 


A- 7 


77.3 


B-5 


84.6 


B-l 3 


84.6 


A- 8 


68.2 


B-6 


69.2 


B-14 


30.3 



A questionnaire concerning the participants 
satisfaction with the presession was administered. This 
questionnaire was kept short in order to avoid burdening 
the participants with too much paper work on top of the 
evaluation form common to all 10 presessions which was 
also administered on the last day. 

This questionnaire with the actual items and 
the responses follows: 

Name (optional) 

Opinionaire about AERA Presession Workshop 
on Human Behavior Genetics , Feb. 25 - Mar. 2. 
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Did you learn as much from the workshop as you thought 
you would when you signed up? 

No 2 Yes 15 More 2 

Would you have participated if you had a chance to 
relive this week? 

No 1 Yes 18 

Were the movies useful? 

Very 3 Yes 16 No 0 

Rate each instructor 1 to 5, with five high. 

Mean Ratings: Scarr 3.9, Henry 2.7, Jensen 4.1, 

Vandenberg 3.7 

Which topics did you find most interesting? 

All topics were mentioned by someone. The topics 
on Monday were especially singled out. 

Are you planning to use what you learned? 

Yes 19 

Was the time schedule satisfactory? 

Yes 17 Days too long 2 

Was the arrangement of the room satisfactory? 

Too crowded 5 Satisfactory 14 

Please comment on any other aspect of the program that 
pleased or displeased you. (The following are comments 
that were made--mostly unsigned). 

1. I would have wanted to spend more time on 
Jensen’s material and subsequent replies. 

2. Fine program. 

3. Was disappointed about the continuing slide 
projector problem. (Some slides did not fit projector.) 

4. More on quantitative methods and data 
treatment and analysis. 

5. Could have used a little more elbow room. 

6. I felt it to be of great value and stimu- 
lation to me personally, especially since I knew rela- 
tively little of the biochemical and population genetics 
aspects . 

7. Excellent workshop!!!! 

8. I liked the informality coupled with the 
unusually convenient opportunities to interact with the 
staff and participants. 
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9. It would be helpful to have fairly struc- 
tured material to read outside class on all topics. 
Reprints of figures and tables would also be helpful. 
I would have been interested in spending more time on 
quantitative genetics. 

10. I think we could use a geneticist — non 
behavioral to blend more of the animal studies with 
human work. 



11. Room problems occurred due to hotel manage- 
ment mix-up. 



12. Air conditioning too loud. 

13. Dr. Scarr was always organized and able to 
explain slides clearly and could anticipate when the 
group was having difficulty with some material. 

Letters received since the workshop contained 
the following statements: 

'I thoroughly enjoyed the workshop and I felt 
it was very informative. It gave me a good idea of 
what is happening in Behavior Genetics today.' Jim Halle 

'I'm afraid that one thing which I learned from 
this year's AERA conference is that it is much better 
to attend a meeting away from one's home community. I 
discovered that it is much too easy to recall the 
pressures which are building up at the office and 
decide not to stay at a particular meeting. All this 
is to say that I attended the Behavior Genetics preses- 
sion only on Thursday and part of Friday. From what I 
did see and hear, however, I learned a great deal and 
appreciate your willingness to develop the preses- 
sion.' Jack E. Rossmann 

'Let me thank you at this time for a most useful 
experience at the Presessions. I am already beginning 
to formulate plans for research in the area of Behavior 
Genetics as a result of what I was introduced to at 
Minneapolis. Keep up the good work.' Conrad A. Reiche rt 

'I would like to state my satisfaction with this 
Presession. It is my personal feeling that it was 
highly informative, well presented and useful to me 
both as a teacher and researcher. ' David R. Taylor 

'I attended every single blessed one of the 
sessions at the workshop on Behavior Genetics and found 
them without exception to be informative, stimulating, 
and very worthwhile.’ L. R. Aiken 
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'The only criticism I have of your Presession is 
that it would have been helpful if I could have received 
the list of references somewhat earlier. I ordered 
the Fuller and Thompson book and your own 1968 'Progress' 
volume ahead of time; however, your book arrived only 
a few days before I left for Minneapolis, while Fuller 
6 Thompson did not arrive at all (it was waiting for 
me when I returned this week).' Jack G. Hutton, Jr. 

'I would like to say again that I have profited 
greatly from your Presession. We have two seminars 
underway here at the Graduate School which will deal 
with the topic of genetics and with Dr. Jensen's work 
in particular. I believe that I can make a far more 
meaningful contribution to these discussions because 
the data presented at the Presession brought the topic 
into focus very clearly. It was my privilege to have 
been a participant.' Pamela Sarett 

'I enjoyed the workshop very much and learned 
a great deal. However, I was somewhat ill during 
the last afternoon session and didn't get as much out 
of it as I would have hoped. f Richard W. Smith 

'I attended all sessions and thoroughly enjoyed 
every minute.' Jacques H. Robinson 

'With time out for church services, I attended 
every session you conducted.' Key T. Lee 

'It certainly was a pleasure to meet you and 
your staff. Needless to say, I thought the Presession 
to be very informative and an exciting learning exper- 
ience. ' Gloria D. Bernheim 

'I must say that the meetings and the subjects 
explored made the training session the best I have 
ever experienced.' John M. Ewing 

'Enjoyed it. Best wishes.' Alexander I. Law 

'I want to express my appreciation for your 
supervision. I enjoyed the sessions and feel that they 
were well worth the time involved.' Suzy E. Holmes 

'You can quote me to AERA sayin g that I have 
attended several workshops and I feel that this was 
the best one in which I have participated. ' Vernon 
Van De Riet 

'I certainly appreciated the opportunity of 
attending the Presessions and feel I gained much from 
it. Tonight I begin teaching a class on the Biology 
of Learning and Behavior for graduate education students. 
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It is the first time that such a course has been offered 
by our college. Most of our topics will be concerned 
with Behavioral Genetics.' Donald L. Lantz 

9. Director's Evaluation! 

From the results of the evaluation, it appears 
that the participants considered the Presession a suc- 
cess and thought that they learned a lot. The objectives 
of the workshop have clearly been met. 

More time between the announcement of the preses- 
sion and the actual workshops would permit more time 
for preliminary reading by participants. 

Holding each Presession by itself, perhaps in 
a motel, would probably solve the problem of an adequate 
meeting room and adjacent bedrooms for the participants. 

Use of handouts which contain the same informa- 
tion as the slides do would give the participants more 
permanent information. 

We could have proceeded somewhat slower if we 
had had another day. 

The local coordinators should be university con- 
nected, preferably an assistant professor who is an 
AERA member. 
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PRESESSION VII 

1. Title: A Systems Approach to Instructional Re- 

search and Design 

2. Staff: Stephen L. Yelon 

(Director) 

Roger 0. Scott 
(Director) 



Norman T. Bell 
Allan J. Abedor 
Frank Cookingham 
Frank A. Smith 



3. General Description: 

During the past few years, a great deal has 
been written about the value of precise, measurable 
educational objectives. Yet, educators are typically 
frustrated in their attempts to develop objectives for 
instructional sequences to be used in their teaching or 
research. Their efforts are often reported as diffi- 
cult and unrewarding and the resulting objectives are 
frequently trivial, unrepresentative or unrealistic. 

It is cur experience that the major cause of 
these problems is the lack of a technology devoted to 
the analysis and derivation of instructional variables. 
The neglect of this area has doomed many applications 
of innovative instruction to failure or only partial 
success. A naivet£ regarding this technology on the 
part of the researchers has led to many studies with 
muddled results — studies based upon curricula which 
do not lead to the desired behaviors, studies which 
were improperly sequenced or studies which did not 
take into consideration important factors influencing 
subject behavior. 

Elements of the technology of the analysis and 
design of instructional variables do exist. Much has 
been written about behavioral objectives and the tech- 
niques of task analysis have been outlined by a num- 
ber of authors. However, acquisition of skills in 
objective writing and task analysis is not sufficient. 
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What is needed is the ability to use a complete strategy 
which leads the educator or researcher from a broad 
general goal to the development of specific plans for 
subject behaviors. This is a prerequisite for useful 
instructional research, for the development of ef- 
fective instruction and for improving the unhappy 
record of educational innovations. 

4. Objectives: 

The following objectives concern the participant's 
abilities to analyze the variables involved in the in- 
structional problem each participant brought to the 
presession. At the end of the presession, participants 
turned in a report which included: 

1. A statement of the problem in form of a 
question, which when answered, will achieve the desired 
instructional goal. 

2. A description of the system including all 
subsystems, their functions and interrelations accor- 
ding to the format given in the presession. 

3. A description of the system as explained by 
a theory or model which best fits the system's cons- 
traints, resources, and goals. 

4. A description of the modifications of the 
system as suggested by the model or theory. 

5. A list of the dependent variables drawn from 
the suggested modification. 

6. A list of the independent variables drawn 
from the suggested modification according to the for- 
mats given in the presession. 

7. Statements showing the relationships between 
the variables listed. 

8. At least one dependent variable stated in 
operational terms according to the format and the 
criteria given ir> the presession. 

9. At least one dependent variable described 
by a flow diagram so that it could be used as a set of 
instructions . 

10. Requisite concepts, principles, facts, 
skills or strategies of dependent variables listed. 

11. At least one independent variable stated 
in operational terms. 
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12. A cost-benefit analysis according to the for 
mula given in the presession. 

5. Schedule: 



First Day: Morning: 

Introduction of staff. Analysis of Pre- 
tests section. Overview. Pretest for 
day's objectives. Presentation: Ob- 

jectives. 

Afternoon: 

Small group work assignments (monitored 
and enforced). Individual Work assignments 

Second Day: Morning: 

In small group, review previous day's 
individual work. In large group, review 
previous day's group work. Pretest day's 
objectives. Presentation: Task Analysis. 



Afternoon: 

Small group work assignments (monitored 
and enforced). Individual work assignments 

Third Day: Morning: 

In small group review previous day's 
individual work. In large group review 
previous day's group work. Pretest day's 
objectives. Presentation: Sequencing 

Strategies. Presentation: Research 

Priorities . 



Afternoon: 

Small group work assignments (monitored 
and enforced). Individual work assignments 

Fourth Day: Morning: 

In small group review previous day's 
individual work. In large group review 
previous day's individual work. Pre- 
test day's objectives. Presentation: 
Materials for Appropriate Practice. 

Afternoon: 

Small group work assignments (monitored 
and enforced). Individual work assignments 

Fifth Day: Discussion and feedback. Evaluation 



6. Participants: 

Approximately 50 individuals applied to the 
AERA Presesoion, A Systems Approach to Instructional 
Research and Design. The ^taff planned to have no more 
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than 30 participants in order to allow an adequate amount 
of monitoring and guidance. For this reason, only the 
first 37 applicants were accepted. As expected, there 
were several last minute withdrawals and a total of 
30 individuals completed the presession. 

7. Instructional and Evaluation Materials: 

A package of instructional materials was sent to 
each participant in advance of the presession. This 
package described the presession objectives and asked 
a series of questions about a description of an instruc- 
tional problem. The responses to these questions gave 
the staff an indication of the entry skill levels of 
participants. During the presession, more than 200 
pages of handouts were made available. These materials, 
many of which were text and learning exercises written 
exclusively for the presession, were packaged in a 
participant notebook. The notebook materials included 
the following units: introducing instructional systems 

(9 pages), writing a problem description (7 pages), 
using educational and psychological models (9 pages), 
making cost/benefit analyses (33 pages) and analyzing 
and operationalizing instructional variables (70 pages). 
In addition to the above materials, each participant 
was asked to work through a programmed text, A Strategy 
for Writing Objectives . 

8. Evaluation and Test Results: 

During the course of the presession, each parti- 
cipant was encouraged to apply the skills being taught 
to an instructional problem which he needed to solve. 

The staff felt that it was important to evaluate the 
presession in terms of how well the skills identified 
inthe objectives were being applied to analyses of 
these problems. Accordingly, twelve criteria, each 
representing the attainment of all or a portion of a 
presession objective, were selected and used to evaluate 
each participant's analysis of his own instructional 
problem. The results of this evaluation show that 
20 or more participants attained at least ten of the 
criteria. The following criteria were used in the 
evaluation: 



1. Is the problem stated in the form of a 
question, which when answered will achieve the desired 
instructional goal? 

2. Is there a statement which includes the 
learner's, teacher's, and the environment's character- 
istics? (At least three descriptors of each.) 

3. Has a theory or model been proposed as a 
foundation for a problem solution? 
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4. Is a change in learner, teacher, and/or 
environmental characteristics proposed as a problem 
solution? 



5. Are dependent variables which define the 
solution or a part of the solution listed? 

6. Are independent variables which define the 
proposed change or a part of the proposed change in 
the learner, teacher, and/or environmental character- 
istics listed? 

7. Is there a statement of the hypothesized 
relationship between at least one dependent and one 
independent variable? 

8. Is at least one dependent variable defined 
in operational terms? (Learner behavior, conditions 
and criteria for acceptance.) 

9. Is there a flow chart illustration of at 
least one dependent variable? 

10. Is there a statement of skills prerequisite 

to the behavioral change stated in the dependent variable? 

11. Is at least one independent variable de- 
fined in operational terms? 

22. Is there a cost estimate for making the 
proposed changes? 

Twenty-six of the 30 presession participants 
responded to an anonymous questionnaire. This instru- 
ment was designed to measure whether the participants 
felt that they had learned skills which the staff had 
attempted to teach and whether they judged these skills 
to be useful. 

9. Director's Evaluation: 

The Program was well received by the participants. 
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PRESESSION VIII 
Title : 



1 

2 



Staff : 



Person-Free Item Calibration and Item- 
Free Person Measurement 



Benjamin Wriglr: 
(Director) 

William Bramble 

Ward Keesling 



William Schmidt 
William Angoff 
General Description: 



University of Chicago 

University of Kentucky 

University of California, 
Los Angeles 

Michigan State University 
Educational Testing Service 



The topic is a mathematical model for objective 
measurement in educational research. We describe, 
explain, and illustrate a new approach to psychological 
measurement which achieves an important kind of measure- 
ment objectivity. The approach is based on the work of 
the Danish mathematician, Georg Rasch. Illustrations 
focus on the dichotomous observations characterizing 
ordinary item analysis. 



The nuclear issue in measurement is the choice 
of model used to transform qualitative observations 
into quantitative measurements. A vital property of 
the model is the extent to which it makes possible 
measurement objectivity. Operationally, objectivity 
means that the calibration of measurement instruments, 
e.g., standardization of tests, is independent of the 
characteristics of the calibrating objects, e.g., the 
sample of persons used for the standardization. It 
also means that measurements made with calibrated in- 
struments are independent of the particular instru- 
ments used to make the measurements, e.g., the set of 
items chosen from the item pool to compose a test. 



The importance of this approach for psychological 
measurement stems from its handling of the connection 
between the normative sample used for test standardiza- 
tion and the subsequent use of the test in measurement. 
The measurement objectivity achieved avoids the usual 
connection between instrument calibration and person 
measurement. Instruments are calibrated and item pools 
are built on a person-free basis so that the unknown 
true distribution of the latent trait becomes irrelevant 
to instrument construction. Once an item pool is con- 
structed, instruments for making individualized person 
measurements can be composed out of any subset of items. 
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Even though each measuring instrument may contain a 
different set of items, the resulting measurements can 
be on a common scale for the latent trait to be measured. 



l ». Objectives: 

Participants acquired an overview of this ap- 
proach to measurement including a recognition of the 
importance of measurement objectivity and an elemen- 
tary mastery of how logistic models are used to solve 
measurement problems. Participants worked through 
applications of the method to real data, and were 
able to apply the method to their own data if they 
wished. They had a chance to work on measurement 
problems during the session. 

5. Schedule: 



First Day: Morning: 

Introduction and orientation. Organiza- 
tion of data analyses. 

Afternoon: 

The Theory of Objective Measurement. 

Set up first round. 

Evening: 

Measurement tutorial. Compute first 
round. 

Second Day: Morning: 

Probability Structure of the Measurement 
Model. Evaluate first round. 



Afternoon: 

Logistic Estimation of Model Parameters. 
Set up second round. 

Evening: 

Data Analysis tutorial. Compute second 
round. 

Third Day: Morning: 

Mean Value Estimation of Model Parameters. 
Afternoon: 

Statistical Control of the Model 

Evening: 

Free 

Fourth i>ay: Morning: 

The Necessity of the Measurement Model. 
Evaluate final round. 
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Afternoon: 

Computer Programming of the Model 

Summarize data analyses. 

Evening: 

Data Analysis tutorial. 

Fifth Day: Morning: 

The Strategy of Item Pools and Test Designs. 

6. Participants: 

The 37 full-time participants attending the AERA 
Presession on 'Person-Free Item Calibration and Item- 
Free Person Measurement' represented a wide range of 
abilities and interests. They represented 13 univer- 
sities, b test publishers, 4 government institutions, 
and 2 state departments of education located in 18 
states and Canada. 

The geographical distributions of those attending 
are as follows: Minnesota, 5 ; California, 4 ; New York, 

4; Canada, 3; Illinois, 3; District of Columbia, 3, 
Maryland, 3; Georgia, 1; Wisconsin, 1; Florida, 1; 

Rhode Island, 1; Virginia, 3; North Carolina, 2; Michi- 
gan, 2; Kentucky, 1; Oklahoma, 1; Ohio, 1; Massachusetts, 
1} and Pennsylvania, 1. 

The universities represented include the 
Universities of California at Los Angeles, Chicago, 
Georgia, Kentucky, Maryland, Miami, Minnesota, Vir- 
ginia, and Wisconsin at Milwaukee; Michigan State, 

New York State, and Ohio State Universities; and 
Rhode Island State College. 

The test publishers include Educational Testing 
Service, Harcourt, Brace and World, Inc., American 
Guidance Service, Ontario Institute for Studies in 
Education, and Educational Services Center. 

The government instutitions include U. S. 

Coast Guard, U. S. Civil Service Commission, Naval 
Medical Research Institute, New York State Department 
of Civil Service, and the California and Pennsylvania 
State Departments of Education. 

7, Instructional and Evaluation Materials: 

Evaluation in such a heterogeneous group is 
difficult. To handle the diversity, a dual scheme 
was adopted. First, participants were asked to define 
their own objectives and reasons for attending. Second, 
four general objectives were specified by the staff 
and participants were asked to indicate which of these 
four were important to them. This information was 
collected at 9:00 a.m. on the first morning of the 
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presession. Then at the end of the fifth morning, 
participants were asked to report how well they felt 
their goals had been realized during the presession. 

8. Evaluation and Test Results: 

A summary of the responses follows. 

Twenty-three participants were interested in 
the application of the measurement model to their own 
data and brought data with them to analyze during the 
presession. One hundred nine computer runs were com- 
pleted for these 23 participants analyzing the appli- 
cation of the measurement model to 39 different educa- 
tional tests. 

Six participants did not complete evaluation forms. 

List the objectives you hope to realize by attending this 
presession. (Tabulation indicates if objective was 
realized. ) 



1. To understand the theory of the measurement 

model . 

Yes, 23 Uncertain, 1 No, 0 

2. To apply the model to real data. 

Yes, 15 Uncertain, >4 No, 1 

3. To clarify the robustness and limitations of 
the model. 

Yes, 9 Uncertain, 1 No, 0 

4. To relate this model to other models. 

Yes, 8 Uncertain, 0 No, 4 

5. To gain mastery of the test construction 
techniques implied by modol. 

Yes, 6 Uncertain, 1 No, 0 

6. To learn how to use computer program for 

model. 

Yes, 4 Uncertain, 2 No, 1 

7. To establish communication with others about 
this kind of work. 

Yes, 3 Uncertain, 0 No, 0 

Additional objectives: 

1. An overview of the logistic model approach 
to measurement. 

Yes, 28 Uncertain, 0 No, 1 
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2. An understanding of the importance of ob- 
jectivity in measurement. 

Yes, 21 Uncertain- 0 No, 0 

3. The ability to apply the model to real data. 
Yes, 24 Uncertain, 1 No, 1 

4. A knowledge of the types of situations in 
which the model can be applied. 

Yes, 17 Uncertain, 8 No, 1 

Comments about various aspects of the presession, with 
number giving various answers: 

1. What did you particularly enjoy about the 
presession? 

11 Style of presentation 
10 Data analysis 
8 Theory 

5 Enthusiasm of staff 

4 Group discussions 

2. What suggestions would you make for the 
improvement of the presession? 

13 Special sessions for advanced participants 

5 More data analysis 
4 More handouts 

3. What were the real strong points of the 
presession? 

9 Data analysis 
7 Theory 
6 Hat.douts 

5 Logical organization 
5 Good speakers 

5 Flexibility and individual attention 

4. What were the weak points? 

6 Too little time 

5 Too mathematical 
4 Boring instruction 
2 Too little data analysis 

5. Were you satisfied with the presession, i.e., 
was it worth the time, money, and effort you expended? 

30 Satisfied 

1 Dissatisfied 

6. Did you analyze data? If so, how did it turn 

out? 

18 Satisfied 

2 Dissatisfied 

“W (Of 23 who brought data> 
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9. Director’s Evaluation: 

The presession was hard but challenging work. 

The variety of participants required preparation of 
materials and presentations on several levels. To 
accomplish this we had a series of preliminary working 
sessions to prepare the instructional staff to deal 
with both naive and sophisticated participants who 
wished to become more sophisticated to follow the 
presentation and to study the presentation material 
subsequently . Finally, we arranged to have simultaneous 
sessions at different levels of sophistication. Even 
so, not all participants were satisfied. A few were 
impatient with elementary session, others were frus- 
trated by sessions which were too advanced for them. 

In spite of these complaints, which were few in number , 
our overall effect war quite satisfying. The majority 
of participants found the presession valuable for them 
as can be seen in the evaluation data. 

Once again, our chief headache was adequate 
communication with a computer. This tim-j we used the 
facilities of Control Data Corporation. We tried to 
arrange to have a remote terminal installed in the 
Hotel Leamington so that we could have immediate access 
to the computer. This turned out to be too expensive 
for us to finance or for Control Data to grant as a 
gift. Instead, we had to rent a car and use valuable 
9taff time driving back and forth between the hotel 
and tb»* computer location. The provision of fresh 
analysis of their ovn data for participants to work 
with and to discuss was and is essential for this kind 
of presession. We were compelled to expend staff 
time to make this possible. But, we look forward to 
the day when several presessions can team up and arrange 
for a remote access terminal located on the site of the 
presessions and connecting all of us with an adequate 
large computer. 

An important outcome of this 1970 presession is 
the continued growth of a group of active test construc- 
tors and psychometricians concerned with the development 
of objective techniques for educational measurement working 
together through their contact during the presession to 
establish better sharing of techniques and experiences 
with each other. This outcome goes beyond the particu- 
lar psychometric techniques taught in the presession 
and extends to the entire professional activity of these 
participants . 
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Dr. Alexander Even of the Ontario Institute for 
Studies in Education continues to coordinate the communi- 
cations of the growing group by publishing a newsletter 
of reports from membei of the group which he distributes 
to members of this group and a larger group of previous 
participants and others interested in objective measurement. 
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PRESESSION IX 



1. Title: 

2. Staff: 



Multivariate Statistical Analysis in 
Educational Research 



M. I. Chc.s. E, Woodson 
(Director) 

R. Darrell Bock 

Neil H. Timm 

Jeremy D. Finn 

Joel R. Levin 
Roger Koehler 
Robert M. Pruzek 



Michael Waller 
Bruno H. Repp 
3. General Description: 



University of Calif- 
ornia, Berkeley 

University of Chicago 

Carnegie Commission 

State University of 
New York at Buffalo 

University of Wisconsin 

Ohio University 

State University of 
New York at Albany 

Chicago, Illinois 

Chicago, Illinois 



The course content consisted of an introduction 
to the concepts and techniques of multivariate analysis, 
including computer programs for making the appropriate 
calculations. Topics included: Hotelling's T , multi- 

variate regression and canonical correlation, multi- 
variate analysis of variance, and multivariate data 
reduction techniques. 



4. Objectives: 

The primary objectives of this presession are: 

1. To present and interrelate the basic tech- 
niques and concepts of multivariate statistical analysis 
and to provide a foundation which will assist tho re- 
searcher in applying these to educational research 
problems. 

2. To assist the participants in gaining some 
practical knowledge of the use of computers for doing 
multivariate analysis calculations. 



S. Schedule: 




First Day: 1. Overview of the Presession and pro- 

cedures. (All instructors.) 
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2 . The Role of Multivariate Methods in 
Educational Research (Bock) 

3. Review of univariate analysis and 
elementary matrix algebra (Woodson) 

4. Elementary matrix operations and more 
matrix algebra with application to data 
matrices TPruzek) 

5. Introduction to computers. Univariate 
examples. One MANOVA example (Finn) 

6. Optional computer session (Finn 6 Levin) 

Second Day: 7. Questions and review of the first day 

8. General multivariate linear model: 
Hotelling’s T* , one- and two-sample (Timm) 

9. Computer exercises (Finn 6 Levin) 

Data reduction techniques: Discriminant 

analysis (Timm) 

10. Intuitive introduction to multivariate 
analysis of variance (Pruzek) 

11. Computer exercises (Finn 8 Levin) 

One- and two-sample repeated measures 

12. Optional computer session (Finn 6 Levin) 

Third Day: 13 Questions and review of the second day 

14. MANOVA: one way (Timm) 

MANOVA: criteria 

15. Computer exercises (Finn 5 Levin) 

16. MANOVA: Factorial designs (Timm) 

17. Data analysis (Pruzek) 

18. Computer exercises (Finn) 

Regression analysis, canonical correlation, 
multivariate analysis of covariance (Timm) 

Fourth Day: 19. Questions and review of the third day. 

20. MANOVA: Unequal cell sizes (Finn) 

21. MANOVA: Nested designs and block 
designs (Timm) 

22. Computer exercises (Finn S Levin) 

The relationship between factor analysis 
and MANOVA (Pruzek) 



O 
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23. 2^ MANOVA designs and planned con- 
trasts (Pruzek) 

24. Computer exercises (Finn 6 Levin) 

Missing data in data reduction problems (Timm) 

25. Optional computer sessions (Finn 6 Levin) 

Fifth Pay: 26. Questions and review of the fourth day. 

27. Methods and issues in the analysis of 
repeated measures and in the analysis of 
covariance (Bock) 

23. MANOVA: Repeated Measures Designs (Finn) 

29. Discussion and evaluation of pre- 
session (Woodson) 

30. Panel discussion: Multivariate 

statistical analysis in educational 
research (all instructors) 

6. Participants: 

Applications were screened and evaluated by the 
director in consultation with the staff. Approximately 
35 applications were received, 75 were admitted and 64 
actually attended. 

A priority system was established on admissions. 
Persons holding a doctorate and teaching graduate 
courses in applied statistics at the level of Hays 
and Kirk were admitted at once. Following that, per- 
sons with a doctorate and reporting considerable formal 
training in applied statistics were admitted. Fol- 
lowing that, parsons without the doctorate but with 
special interest in these techniques were admitted. 

Persons indicating very little background in statis- 
tics were not admitted. 

7. Instructional and Evaluation Materials: 

Handouts were given to all participants. 

8. Evaluation and Test Results: 

None were made . 

9. Director’s Evaluation: 

The major factor in the success of the presession 
on 'Multivariate Statistical Analysis' was the excellent 
staff. Their work and high degree of competence com- 
bined favorably with their variety of interests and 
approaches . 
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One of the major problems we expected and did 
encounter is that many researchers do not know what 
multivariate statistics are and are unprepared to find 
the topic difficult. 

Some participants felt the description in the 
Educational Researcher was misleading. The description 
of the course as teing for applied researchers was rela- 
tively accurate, but more effort could have been made 
to indicate background necessary for multivariate 
statistics . 

The computer sessions were generally disappoin- 
ting. Through a series of difficulties with the com- 
puting facilities available (both the University of 
Chicago facility and the backup commercial facility 
arranged for), actual computer runs were very limited. 
Fortuntely, the staff, particularly Levin and Finn, 
had prepared pre-run outputs which were distributed 
■and discussed. This substitute was effective. This 
experience suggests that in the future we should be 
careful to prepare examples for distribution and not 
count on computer availability. The opportunity to 
submit computer runs is an important part of the ob- 
jectives of the sessions and we hope it will be move 
successful next year. 
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PRESESSION X 

1. Title: Powerful Statistical Techniques for Quali- 

tative as Well as Qualitative Variables 

2. Staff: Douglas A. Penfield Rutgers University 

(Director) 

Maryellen McSweeney Michigan State Uriversity 

(Director) 

Andrew Porter Michigan State University 

Leonard A. Marascuilo University of California, 

Berkeley 

Stephen Lawton University of California, 

Berkeley 

David Wright Michigan State University 

3. General DtJcription: 

Our purpose in teaching this five-day presession 
was to present the most current and the potentially most 
useful nonparametric statistical procedures that could 
be applied in educational research. It was our hope 
that the researchers who attended this session could 
put these techniques to immediate use when analyzing 
their own experimental research. Since more and more 
educators are becoming involved in research and since 
nonparametric methods are increasing in number and in 
versatility of application, it is essential that active 
educators and behavioral researchers be informed and 
kept up to date on advances made in this important 
area of research methodology. 

Our intent was to plan each session so that 
there would be adequate time to cover the topic at hand 
with a minimum of stress on the participants. This 
was handled by scheduling three ninety minute sessions 
each day with a sixty minute informal discussion period 
at the conclusion of the day's lectures. We restricted 
outselves to those nonparametric techniques which we 
believed to be potentially the most important and useful 
to researchers in education and the behavioral sciences. 

We realized that researchers, no matter how diligent, 
could not achieve complete mastery of these techniques 
within the limited time period of the presession. We 
tried to introduce our participants to a wide variety 
of techniques during the presession and to provide 
them with extensive supporting materials to enable them 
to continue their independent study in nonparametric 
methods when the presession was completed. 
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In all cases the discussion and the use of tests 
and confidence intervals focused on applications. The 
mathematical development and proving of theorems and 
related results were not covered as a part of the formal 
lecture periods. These theoretical developments 
appeared as separate sections in the handout materials 
distributed to the presession participants. 

M . Objectives: 

Although this presession was primarily con- 
cerned with statistical techniques for the analysis of 
qualitative and quantitative variable, it was recog- 
nized that experimental design considerations will 
affect the participants' ability: 

a. to provide interpretable estimates of treat- 
ment effects; 

b. to employ precise statistical techniques in 
data analysis. 

For this reason, the informal discussion periods 
were used for more extensive consideration of the 
design problems associated with the statistical tech- 
niques presented. To stimulate discussion of research 
design considerations and of nonparametric analysis 
procedures, many detailed examples of studies requiring 
a nonparametric analysis were presented as separate 
handouts . 

It was hoped that exposure to these procedures 
would aid the participant in solving problems dealing 
with experimental design, hypothesis testing, and esti- 
mation commonly found in educational research. Specifi- 
cally, it was our expectation that, as a result of 
participation in the presession and subsequent inde- 
pendent study of the presession materials, the partici- 
pants would be able to: 

a. Explain the rationale behind the test proce- 
dures presented. 

b. Perform these tests on data from educational 
research. 



c. Apply and extend the techniques learned to 
specific problems in educational research. 

d. Distinguish between experimental designs 
in which variables are confounded and designs which 
provide interpretable estimates of treatment effects. 

3. Employ 'precision' as one criterion in 
selecting an appropriate experimental design and the 
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'best' statistical test for a specific situation. 

f. Read current literature involving methods 
described in this presession. 



g. Direct other researchers in the use of 
these statistical techniques. 



S. Schedule: 

First Day: Morning: 

Discussion of class objectives and a com- 
plete overview of the material to be 
covered during the presession (Penfield). 
Irwin-Fisher exact test (McSweeney). 

Afternoon: 

Median test and Chi-square test of homo- 
geneity (Wright). 

Informal discussion 

Second Day: Morning: 

Extension of the median test to K samples 
(Penfield) . 

Confidence interval procedures for the Chi- 
square and median tests (McSweeney). 

Afternoon : 

Tests of independence in contingency 
tables (Porter). 

Informal discussion. 

Third Day: Morning: 

Test of interaction across contingency 
tables and associated confidence interval 
procedures (Marascuilo) . 

The Wilcoxon test for matched-pairs and 
Spearman* rho (Porter). 

Afternoon : 

Friedman test and Kendall’s coefficient 
of concordance (Porter). 

Informal discussion. 



Fourth Day: Morning: 

Cochran's Q test and Kendall's tau (Lawton) 
Mann-Whitney test (Penfield). 

Afternoon: 

Application of the Wilcoxon test to block 
designs and the test on aligned observa- 
tions (McSweeney). 

Informal discussion. 

Fifth Day: Mroning: 

Normal scores test (Penfield). 
Kruskal-Wallis test (Porter). 

O 
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Afternoon: 

A complete review of all material 
covered (McSweeney). 

Informal discussion. 

6. Participants: 

A total of 52 individuals were accepted for the 
presession, but only 30 arrived for active participation. 
For the 22 who did not attend, the most common reasons 
for failure to participate were lack of funds for travel 
and lodging and insufficient time upon receiving an 
acceptance notice to make the necessary arrangements. 

As was true in previous years, the majority of 
the participants were males. Of the 30 people attending, 
only three were females. In our past two presessions, 
there has always been a larger proportion of males, but 
the dichotomy has never been quite this extreme. 

This year a much larger percentage of the partici- 
pants were in the 'under 30' category. There appears to 
be two major reasons for this large percentage of 
younger participants: 1. Professional staffs are 

acquiring younger members who find they need an upgrading 
of their basic statistical skills, and; 2. Many schools 
do not offer a course in nonparametric statistics and 
advanced graduate students find the presessions an ex- 
cellent opportunity for acquiring needed information. 

It can also be pointed out that as a group, the partici- 
pants were much better prepared to handle the material 
being presented than previous groups had been. This 
was especially true of the people over 40 years of 
age. We have always found the younger individuals to 
be very enthusiastic and excited over the practical 
use they derive from the lectures and discussion sec- 
tions . 



Most individuals hold a position at a university 
or college. There is still an interest in nonparametric 
statistics from individuals employed by research 
centers across the country, but due to cutbacks in 
federal funding only a limited number of resident staff 
members participated in the presession. A number of 
letters and telephone calls were received indicating a 
desire to attend, but due to a curtailment of funds 
it war deemed impossible. 

Most participants were attending an AERA pre- 
session for the first time. Only six out of the 
thirty had attended a session previously and two of 
these six had attended our nonparametric session in 
1968. It was hoped that the spacing of the lectures 
would reduce tensions that might exist for those attending 
their initial presession. 
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Most of the participants are either just completing 
their degree or are recent graduates in their first 
three years of completion. The whole purpose of the 
presessions is to provide an opportunity for people who 
have been out of graduate school for a number of years 
to learn about new innovations in educational research. 
Obviously, this presession is not drawing from the older 
audience. Many older professors have moved into adminis- 
trative positions and find little need for the tech- 
niques we are offering. In order to be competitive, 
however, the younger faculty must make a concerted ef- 
fort to acquire skills conducive to good educational 
research . 

The level of statistical competency as measured 
by the number of previously taken statistics courses 
was higher in 1970 than for either of the two pre- 
vious years this course was taught. This was mentioned 
previously and is a real tribute to the graduate pro- 
grams from which these individuals are graduating. They 
proved to be very alert and knowledgeable about ele- 
mentary statistics . This means that they are able to 
derive much more benefit from the material being pre- 
sented while allowing the instructors greater freedom 
in their presentations. 

The majority of participants have the greatest 
proportion of their time devoted to research. Over 
60 percent of the people are spending more than 50 per- 
cent of their time on research activities. This result 
is similar to what was found to be true in 1968 and 
1969. This year's group, however, is spending much 
less time in the classroom than previous years’ groups. 
Thus, the material covered in this presession will be 
essential for them in their research endeavors. 

It is interesting to note that most of the par- 
ticipants actually have published very little to date. 

This is somewhat related to the fact that they are 
young or graduate students or have only recently fin- 
ished their degrees. The profile of percentages con- 
forms to the pattern set over the past two years. 

Perhaps those who do publish extensively already pos- 
sess a knowledge of the material being presented and 
thus have little desire to attend. It is hoped that 
in future years the participants will use the tech- 
niques learned at this presession to develop their own 
research activities. 

Five people were from the East, seven from the 
South, ten from the Midwest and six from the West. It 
was expected that the draw would be small outside the 
Midwest region. The East, however, was a disappoint- 
ment with only five participants. Certainly the prox- 
imity to the location of the presession is an important 
factor in attendance. 
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7. Instructional and Evaluation Materials: 

Detailed lectures and handouts describing studies 
which employed nonparametric analyses were prepared and 
distributed to each participant. They were fitted into 
a handsome looseleaf notebook costing all of $.79. 
Participants could refer to the multilithed lecture 
notes as the instructor lectured. This allowed for a 
minimum of blackboard writing. Since we have always 
been plagued by a lack of board space, having the 
lectures written up provided a valuable tool for effec- 
tive presentation of material. It was also true that 
the lighting in the classroom left something to be 
desired so the printed lectures were even more ap- 
preciated. 

The printed lectures were somewhat more thorough 
in coverage than the oral presentations since they 
included derivations of the test statistics, and an- 
notated references as well as an 'intuitive' development 
of the techniques and examples of their use. The more 
extensive presentation provided by the written lecture 
notes was thought desirable since the participants would 
have permanent copies of the presession materials to 
be reviewed at their leisure. That the participants 
shared this view of the usefulness of the materials was 
indicated by comments of several participants on the 
final evaluation forms: 

"The printed materials should prove to be a most 
valuable resource when I return to my lab. They were 
tremendously valuable as a follow-up each evening as 
I reviewed the day's work." 

"The amount and intensity of material presenta- 
tion can be assimilated best by reading the well-pre- 
pared mimeo lecture material in the next month or so. 

It needs a while to soak in." 

"The pace was quite rapid, but the overview 
was such that there is a sound base for getting con- 
ceptual and computational mastery at a later time." 

"Vast quantities of very complex material were 
presented. I spent a portion of each evening studying. 

I feel this was appropriate and would have felt cheated 
if the session had been less demanding. Handout mater- 
ials (approximately 300 pages) were most valuable..." 

8. Evaluation and Test Results: 

At the conclusion of the day's lectures, each 
participant was given a multiple choice test covering 
the day's material. This was turned in the following 
morning, corrected, and returned to the participant. 
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A total of four tests were given and corrected. It 
was interesting to note that a number of the older 
participants failed to hand in their tests. No ef- 
fort was made to collect the exams since this would 
only have caused considerable embarrassment to the 
individuals and created an uneasy situation for the 
rest of the session. The statistics for the four tests 
are as follows: 

Test Mean Number Correct Maximum Score Possible 



1 


6 


7 


2 


10 


12 


3 


11 


16 


4 


10 


15 



The staff was exceedingly pleased with the per- 
formance of individuals on the exams. The majority of 
the participants made a concerted effort to attempt 
the questions and thus consolidate the learning that 
had taken place during the day, 

9. Director’s Evaluation: 

At the conclusion of this presession a ques- 
tionnaire developed by the coordinator of the AERA 
Presessions in cooperation with the directors of the 
individual sessions was distributed to each participant. 
The responses of Ihe participants were generally quite 
favorable. Further comments added by some of the 
participants were supportive of the efforts of the pre- 
session staff and included valuable suggestions for the 
improvement of the presession. Particularly promising 
suggestions seemed to be: 

1. The development of a listing of computer 
programs in nonparametric statistics that would be 
available to the participants. 

2. The formation of small study groups to re- 
inforce lectures and larger group discussions. 

3. Greater specificity in indicating the back- 
ground material to be reviewed by the participants 
prior to presession attendance. 

Since the conclusion of the presession, two 
participants have written to express their thanks for 
help given at the presession and to indicate to the 
staff that they are using the techniques taught at the 
presession. 

"Your lectures, instruction and lecture notes 
gave me a much clearer understanding of both parametric 
and nonparametric statistical procedures than I thought 
possible. . . " 
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"We had a show and tell session here at the 
Bureau [of research) yesterday. I gave a brief summary 
of our session and presented transparencies of the out- 
lines of the various tests. I think it was the first 
time many of our professional staff realized that 
there were post hoc procedures available for nonpara- 
metrics. Hodges-Lehmann Test, Lancaster Procedures 
and Normal Scores Tests really impressed the staff, 
and were apparently new to them." 

It was particularly gratifying to the staff to note 
that all of the participants responding to the final 
evaluation felt that they would "use what [they had] 
learned in [their] own teaching or research in the 
immediate future [and] at a later date in future re- 
search or teaching." 

In a recent Review of Educational Research 
article it was observed that, "No matter how powerful 
the newer nonparametric techniques are, their use- 
fulness will depend ultimately on the diligence of 
applied statisticians in explaining them and on the 
receptivity of behavioral researchers to these 
methodological advances." We believe that the con- 
scientious efforts of the participants of this pre- 
session, their avid interest in research methodology, 
and their eagerness to update their knowledge of 
research design and statistical analysis create the 
expectation that the participants will indeed employ 
what they have learned at this presession in their 
future professional activities. The fulfillment of 
such an expectation would amply repay the efforts of 
the staff in preparing and presenting this presession. 

The presession Powerful Statistical Techniques 
for Qualitative as well as Quantitative Variables was 
held in Minneapolis at the Pick-Nicollet Hotel from 
February 26 to March 2, 1970. Thirty people attended. 
Three lectures and one discussion session were held each 
day over a total of five days. The participants appeared 
to be well prepared in elementary statistical tech- 
niques and seemed to benefit greatly from the material 
being presented. Many said it was one of the most 
worthwhile set of five days they had spent in some time. 
Lecture notes were distributed to each participant for 
use at the presession and for future reference. It 
was hoped by the staff that the exposure of the par- 
ticipants to these nonparametric techniques would 
make them better researchers in che years to come. 




